Access through an unexported type

[⚛]

I was trying to search the forum and the language specification for a
description of a particular problem, but wasn't able to find it -
although I remember that I have seen a related problem somewhere on
golang-nuts. The source code for the problem is printed below. The
idea is that *none* of the three cases in "main()" should compile (at
least that is what I would expect), but two of them do compile.

=================
package p

type private interface {
Func()
}

type s struct {}
func (s *s) Func() { println("s.Func()") }

func GetPrivate() private {
return &s{}
}

=================
package main

import "p"

func main() {
// Compiles
p.GetPrivate().Func()

// Compiles
s1 := p.GetPrivate()
s1.Func()

// Does not compile
var s2 p.private = p.GetPrivate()
s2.Func()
}

[Russ Cox]

The export rules affect whether a name in one package
can be used to refer to a name in another package.

In the examples in your program, the first two do not
use the name "private" so they are allowed.

Russ

[Steven]

Private is a property of the identifier. If an identifier is private, you can't refer to it outside the package. That doesn't mean that you can't use something that is private:

var i int

var P = &i

func private() {}

var Public = private

A type being private means you can't refer to the type in your code. It doesn't mean you can't use a value of that type that is given to you. Consider:

type privateType int

var PublicVar privateType

There's no reason, in the spec or otherwise, you wouldn't be able to use PublicVar outside the package.

[peterGo]

Exported identifiers
http://golang.org/doc/go_spec.html#Exported_identifiers

The package p identifiers GetPrivate and Func are exported. The
package p identifier private is not exported. In the package main you
can only use package p identifiers that are exported. Therefore, the
first two cases compile and the third case doesn't compile (error
message "cannot refer to unexported name p.private").

Peter

[⚛]

About the 2nd case vs. the 3rd case in the source code I provided: The
section [http://golang.org/doc/
go_spec.html#Short_variable_declarations] says that "[short variable
declaration] is a shorthand for a regular variable declaration ...".
That suggests that we are talking about 1:1 equivalence. That is,
properties of "var v t = expression" are exactly the same as the
properties of "v := expression" in all possible contexts in which the
language allows the usage of both of these two forms.

Similarly, in relation to the 1st case "p.GetPrivate().Func()", I
would say that [a programming language in which this expression
compiles is consistent] if the expression *always* can be replaced
(when used in e.g. "main()") with something like this:

func get_it() p.private { return p.GetPrivate() }
func main {
get_it().Func()
}

But, unfortunately, the subexpression "p.GetPrivate()" cannot be
encapsulated into a new function because it is impossible to define
its return type.

[Russ Cox]

> That suggests that we are talking about 1:1 equivalence.

We're not.

There are other contexts in which they are different too,
such as the behavior when there are multiple variables
being declared.

> unfortunately, the subexpression "p.GetPrivate()" cannot be
> encapsulated into a new function because it is impossible to define
> its return type.

Yes. You should talk to the person who wrote that API.

Russ

[⚛]

On Mar 11, 8:22 pm, Russ Cox <r...@google.com> wrote:
> > That suggests that we are talking about 1:1 equivalence.
>
> We're not.

Well, but what does "shorthand" stand for? Implication (A=>B), or
equivalence (A=B), or something else? I think "shorthand" isn't a
mathematical term. Shouldn't the language specification better choose
between the following two:

- Implication: If the compiler compiles "var v t = expression", then
it compiles "v := expression".

- Equivalence: The compiler compiles "var v t = expression" if and
only if it compiles "v := expression".

> There are other contexts in which they are different too,
> such as the behavior when there are multiple variables
> being declared.
>
> > unfortunately, the subexpression "p.GetPrivate()" cannot be
> > encapsulated into a new function because it is impossible to define
> > its return type.
>
> Yes.  You should talk to the person who wrote that API.
>
> Russ

Well, I found this "API" in the code of a project that I am
participating in. So, that would mean that I should talk to myself ...
and myself thinks that it would be good to put the blame on the
compiler for allowing me to write such nonsense ...

[⚛]

On Mar 11, 8:11 pm, peterGo <go.peter...@gmail.com> wrote:
> Exported identifiershttp://golang.org/doc/go_spec.html#Exported_identifiers

>
> The package p identifiers GetPrivate and Func are exported. The
> package p identifier private is not exported. In the package main you
> can only use package p identifiers that are exported. Therefore, the
> first two cases compile and the third case doesn't compile (error
> message "cannot refer to unexported name p.private").
>
> Peter

I do understand why the first two cases compile and the 3rd does not.
What I do not understand is: why is the language built like that? As I
explained in my second post in this forum thread, the language looses
some of its orthogonality.

[Steven]

Out to fight the world are we?

On Fri, Mar 11, 2011 at 3:40 PM, ⚛ <0xe2.0x...@gmail.com> wrote:

On Mar 11, 8:22 pm, Russ Cox <r...@google.com> wrote:
> > That suggests that we are talking about 1:1 equivalence.
>
> We're not.

Well, but what does "shorthand" stand for? Implication (A=>B), or
equivalence (A=B), or something else? I think "shorthand" isn't a
mathematical term. Shouldn't the language specification better choose
between the following two:

- Implication: If the compiler compiles "var v t = expression", then
it compiles "v := expression".

- Equivalence: The compiler compiles "var v t = expression" if and
only if it compiles "v := expression".

The spec says `x := expression` is shorthand for `var x = expression` (not `var x T = expression`), with the added ability that it can redeclare variables in the same scope, as long as it is still declaring at least one variable. 

Both work in this case. Please reread the spec for shorthand declaration.

 

> There are other contexts in which they are different too,
> such as the behavior when there are multiple variables
> being declared.
>
> > unfortunately, the subexpression "p.GetPrivate()" cannot be
> > encapsulated into a new function because it is impossible to define
> > its return type.
>
> Yes.  You should talk to the person who wrote that API.
>
> Russ

Well, I found this "API" in the code of a project that I am
participating in. So, that would mean that I should talk to myself ...
and myself thinks that it would be good to put the blame on the
compiler for allowing me to write such nonsense ...

The compiler shouldn't do anything to disallow something that is allowed in the spec. And there's no reason for the spec to specifically disallow a behaviour that follows directly from both the rules and the implementation without extra work. If the behaviour isn't useful to you, don't use it. However, it is conceivable someone would desire the restrictions of making values of an unexported type available, such as to prevent dependency on a specific type in the public interfaces of other packages.

[John Asmuth]

On Mar 11, 3:40 pm, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
> On Mar 11, 8:22 pm, Russ Cox <r...@google.com> wrote:

> > Yes.  You should talk to the person who wrote that API.
>

> Well, I found this "API" in the code of a project that I am
> participating in. So, that would mean that I should talk to myself ...

http://liverpool.theoffside.com/files/2010/10/thats_the_joke.jpg

[⚛]

On Mar 11, 8:48 pm, Steven <steven...@gmail.com> wrote:
> The spec says `x := expression` is shorthand for `var x = expression` (not
> `var x T = expression`), with the added ability that it can redeclare
> variables in the same scope, as long as it is still declaring at least one
> variable.
>
> Both work in this case. Please reread the spec for shorthand declaration.

Are you sure it is not you who should reread it?

The spec also says: "If the type is present, each variable is given
that type. Otherwise, the types are *deduced* from the assignment of
the expression list."

So, if the types are deduced, I am *not* accessing them and therefore
I am implicitly allowed to use unexported types there?

What does "deduced" stand for? I think it stands for "the types are
filled in automatically". This means that "var v = expression" should
be the exact same thing as "var v t = expression", assuming the
expression evaluates to type "t".

> The compiler shouldn't do anything to disallow something that is allowed in
> the spec. And there's no reason for the spec to specifically disallow a
> behaviour that follows directly from both the rules and the implementation
> without extra work. If the behaviour isn't useful to you, don't use it.

Why don't you explain to me what "deduced" and "shorthand" stand for?

[Russ Cox]

You are conflating C++ and Java ideas of public and private
with the Go idea of exported. Exported is about which names
one package can use to refer to things (types, methods, struct
fields, functions, variables) in another package. Go's exported
is a purely lexical concept. That's why it is so easy to understand
and explain. That has consequences that end up being different
from what public and private mean in C++ or Java, but they are
still easy to explain. That is why the spec is what it is.

Russ

[Steven]

Okay, so you are out to fight the world. You might have more luck making ice cream. I've always wanted to try making ice cream...

On Fri, Mar 11, 2011 at 4:12 PM, ⚛ <0xe2.0x...@gmail.com> wrote:

On Mar 11, 8:48 pm, Steven <steven...@gmail.com> wrote:
> The spec says `x := expression` is shorthand for `var x = expression` (not
> `var x T = expression`), with the added ability that it can redeclare
> variables in the same scope, as long as it is still declaring at least one
> variable.
>
> Both work in this case. Please reread the spec for shorthand declaration.

Are you sure it is not you who should reread it?

The spec also says: "If the type is present, each variable is given
that type. Otherwise, the types are *deduced* from the assignment of
the expression list."

So, if the types are deduced, I am *not* accessing them and therefore
I am implicitly allowed to use unexported types there?

What does "deduced" stand for? I think it stands for "the types are
filled in automatically". This means that "var v = expression" should
be the exact same thing as "var v t = expression", assuming the
expression evaluates to type "t".

Deduced doesn't stand for anything. It just means deduced. Any equivalence you've set up in your mind is artificial. The type of the value is deduced. This does not mean that the compiler fills in the "type" slot in the expression for you. This doesn't make sense, since you there is no type slot in a multiple type, multiple variable declaration/assignment.

Shorthand doesn't stand for anything. It means shorthand, as in, its a shorter way of writing the same thing. The spec also says that it is shorthand, except for a specific difference, which means that it is in all aspects the same of what it is shorthand for except one.

Your argument is based on interpreting words differently from the way that the authors of those words interpreted them. They also depend on interpreting words in a non-standard way based on your notion of what they should mean in that context. This is one step  (or to be fair, maybe two steps) away from saying "it should be this way because I think so, and no amount of rational argument will dissuade me".

> The compiler shouldn't do anything to disallow something that is allowed in
> the spec. And there's no reason for the spec to specifically disallow a
> behaviour that follows directly from both the rules and the implementation
> without extra work. If the behaviour isn't useful to you, don't use it.

Why don't you explain to me what "deduced" and "shorthand" stand for?

Considering you asked your question after I had written that paragraph, your continued insistence is premature. 

[yy]

2011/3/11 ⚛ <0xe2.0x...@gmail.com>:

> func get_it() p.private { return p.GetPrivate() }
> func main {
>    get_it().Func()
> }
>
> But, unfortunately, the subexpression "p.GetPrivate()" cannot be
> encapsulated into a new function because it is impossible to define
> its return type.

It is not impossible, what is impossible is to refer to p.private
because is not exported, but this will work:

func get_it() interface{Func()} { return p.GetPrivate() }

It can look a bit strange, but it would look much better if somewhere
(main, or p, or somewhere else) you had defined an interface:

type Funcer interface{Func()}

and GetPrivate was called GetFuncer.

This can be useful in some cases. For example, see the encoding binary
package, where the types littleEndian and bigEndian implement the
ByteOrder interface but the package documentation is much clearer not
including all their methods.

--
- yiyus || JGL .

[⚛]

On Mar 11, 9:21 pm, Steven <steven...@gmail.com> wrote:
> Deduced doesn't stand for anything. It just means deduced.

Brilliant

[Steven]

Thank you. I pride myself on my astutitude.

You asked what it stood for. It doesn't stand for anything, its a word in its own right. Do you want a definition?

Deduce:
To reach a conclusion by applying rules of logic to given premises.

All the statement says is that the type of the variable is deduced, not that its identifier is somehow inserted into the text of the program. There isn't even a place where they type's identifier could be consistently inserted into the text of the program.

Its isn't the types that are exported or not, it is their identifiers. As long as you don't use the identifier (ie. the actual type itself is deduced without needing the identifier), then you are fine.

Need any more help? I'm good with dictionaries.

[⚛]

I thought that I would write a lengthier text, but I have decided not
to, since the probability of it landing in the language would be
really low since I have no actual power over the language
specification. But, at least, I would like to note that it is *not*
true that "Go's exported is a purely lexical concept". If it was a
purely lexical concept, you would be able to compile the following
code:

==========
package p
type T struct { a int }
func MakeT() T { return T{} }

==========
package main
import "p"
func main() {

a := p.MakeT() // Error
println(&a)
}
==========

There is no lexical occurrence of "p.T" in the source code below
"package main". The fact that line "a := p.MakeT()" fails to compile
indicates that Go's notion of "exported" is not a purely lexical
concept.

On Mar 11, 9:17 pm, Russ Cox <r...@golang.org> wrote:

[⚛]

On Mar 11, 9:24 pm, yy <yiyu....@gmail.com> wrote:
> This can be useful in some cases. For example, see the encoding binary
> package, where the types littleEndian and bigEndian implement the
> ByteOrder interface but the package documentation is much clearer not
> including all their methods.

Thanks for pointing it out.

[Steven]

On Sat, Mar 12, 2011 at 5:12 AM, ⚛ <0xe2.0x...@gmail.com> wrote:

I thought that I would write a lengthier text, but I have decided not
to, since the probability of it landing in the language would be
really low since I have no actual power over the language
specification. But, at least, I would like to note that it is *not*
true that "Go's exported is a purely lexical concept". If it was a
purely lexical concept, you would be able to compile the following
code:

==========
package p
type T struct { a int }
func MakeT() T { return T{} }

==========
package main
import "p"
func main() {

       a := p.MakeT()   // Error
       println(&a)
 }
==========

There is no lexical occurrence of "p.T" in the source code below
"package main". The fact that line "a := p.MakeT()" fails to compile
indicates that Go's notion of "exported" is not a purely lexical
concept.

Exported vs non-exported is defined here:

http://golang.org/doc/go_spec.html#Exported_identifiers

As you can see, it is a property of identifiers, not of the things identified.

What you are observing is the result of a special case rule. Structs containing non-exported fields are, as a whole, not assignable outside their packages of origin. Another special rule for struct fields is that you can't give them a value in a composite literal. Note that neither of these rules stem from the definition of exported vs non-exported, but rather from special properties of assignments and composite literals. If someone gives you a pointer to a non-exported field, you can assign to the field as much as you want via the pointer.

[⚛]

You wrote: "Structs containing *non-exported* fields are, as a whole,

not assignable outside their packages of origin."

Immediately then you wrote: "Note that *neither* of these rules stem
from the definition of *exported* vs *non-exported* ..."

I am confused. You consider this to be sound reasoning? Was that
supposed to be some joke?

On Mar 12, 2:39 pm, Steven <steven...@gmail.com> wrote:

[chris dollin]

On 12 March 2011 15:49, ⚛ <0xe2.0x...@gmail.com> wrote:
> You wrote: "Structs containing *non-exported* fields are, as a whole,
> not assignable outside their packages of origin."
>
> Immediately then you wrote: "Note that *neither* of these rules stem
> from the definition of *exported* vs *non-exported* ..."
>
> I am confused. You consider this to be sound reasoning?

I don't see the problem. Yes, neither of the two rules stem from the
/definition/ of exported vs non-exported. They're extra rules which
are not logical consequences of that definition.

> Was that supposed to be some joke?

It doesn't look like a joke to me.

Chris

--
Chris "allusive" Dollin

[Russ Cox]

On Sat, Mar 12, 2011 at 05:12, ⚛ <0xe2.0x...@gmail.com> wrote:
> I would like to note that it is *not*
> true that "Go's exported is a purely lexical concept". If it was a
> purely lexical concept, you would be able to compile the following
> code:
>
> ==========
> package p
> type T struct { a int }
> func MakeT() T { return T{} }
> ==========
> package main
> import "p"
> func main() {
>        a := p.MakeT()   // Error
>        println(&a)
>  }
> ==========

Yes, you're right. The copying restriction is not based in the
lexical rule. It is a separate add-on. Maybe we should drop it.

Russ

[⚛]

On Mar 12, 4:41 pm, chris dollin <ehog.he...@googlemail.com> wrote:

> On 12 March 2011 15:49, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
>
> > You wrote: "Structs containing *non-exported* fields are, as a whole,
> > not assignable outside their packages of origin."
>
> > Immediately then you wrote: "Note that *neither* of these rules stem
> > from the definition of *exported* vs *non-exported* ..."
>
> > I am confused. You consider this to be sound reasoning?
>
> I don't see the problem. Yes, neither of the two rules stem from the
> /definition/ of exported vs non-exported. They're extra rules which
> are not logical consequences of that definition.

I don't agree. The primary reason for the existence of sentence
"Structs containing non-exported fields are, as a whole, not
assignable outside their packages of origin" is the notion of non-
exported versus exported. Let's call this sentence "rule 1".

The idea that you (and Steven) are presenting is that the mentioned
sentence is "just an extra rule". If it was just an extra/optional
rule, then it could just simply be dropped from the language. And this
is where you are wrong: you cannot drop it. The reason why it cannot
be dropped is that the question "What do you do about assignment when
a structure contains non-exported fields?" exists *because* fields of
a structure can be exported or non-exported. Once the fields can have
this kind of a property, that question *must* be answered by the
language designer. Of course, there are *multiple answers* to this
question, and it is up to the language designer to decide which answer
(rule) is the best one. Of course, the language designer can refuse to
answer the question - but the question will be answered nevertheless
(the answer will be an indirect consequence of the language
implementation).

If you drop "rule 1" from the language, then *inevitably* you will
introduce some other rule to the language. Contrary to your opinion,
the rule "rule 1" does stem from the /definition/ of exported vs non-
exported. But of course, you can change the language and use some
other rule instead. The point is that you have to choose *some* rule,
you do not have the choice not to use any rule.

If you completely remove the concept of "exported/non-exported struct
fields" from the language, the sentence "rule 1" will loose all
reasons for its existence. This is why it stems from the concept of
exported/non-exported.

It does matter whether you choose "rule 1" or some other rule. The
consequences of either choice are non-trivial. And please recall from
the previous paragraphs: you don't have the choice not to make any
choice.

[Steven]

Are you referring to the discussion or the copying restriction? 

If the latter, then I have mixed feelings on that. The copying restriction gets in the way of implementing some things, such as anything slice-like, or immutable points, ect. On the other hand, it allows you to force people to use your type via a pointer for when you need to preserve invariants. I'm not sure non-exported fields should be the mechanism for this, since it doesn't match up with what the concept means, and certainly it isn't universally true that you shouldn't be able to copy a struct with non-exported fields.

If the restriction were removed, I would still like to see the ability to make it an error to copy a value of a certain struct type. Its a simple enough error that could be hard to track if you don't realize what the mistake is. I guess the mistake would show up when the library expects pointers and you're trying to use values in your code. But a lot of intelligent people have trouble with pointers, and may just take an address locally without realizing the implication that they are supposed to always use a pointer.

Certainly, its possible to do something like:

struct myFragileType struct { ... }

struct PointerToMyFragileType struct { *myFragiletype }

But that prevents people from embedding your type by value and then using a pointer to their own struct, which is a useful optimization. I guess you can't save people from everything without restricting desirable behaviour, and as long as people read the documentation for your type, they will know they are supposed to use a pointer.

This reminds me of the properties discussion I had with Ellie a while back:

http://groups.google.com/group/golang-nuts/browse_thread/thread/68defff7c2198503/d38362fb027f59f5?lnk=gst&q=properties#d38362fb027f59f5

[Steven]

I disagree with your premise that the existence on non-exported fields requires that the rule, or some other rule like it, exists. All the definition of exported (and non-exported, by implication) says is that you can use exported identifiers from other packages via a qualified identifier (and by implication, that you can't do this with non-exported identifiers). It doesn't say anything about any protection. So the rule could be dropped, and your question would be irrelevant. There's nothing in the spec aside from that rule that implies that you can't indirectly access something that has a non-exported identifier from another package. If you think that the whole point of non-exported identifiers is prevent this, then you are confusing it with the idea of "private" from other languages.

The rule is there to protect invariants in non-exported fields. There are other ways of doing this, such as with a keyword or a runtime flag saying "others can't copy values of this struct type", or even just putting "always use this type via a pointer" in the documentation for the type. As I said in my reply to Russ, non-exported fields in a struct do not necessarily mean there is an invariant that needs to be protected.

[⚛]

May I ask you a question: Have you ever developed your own programming
language, or participated in a group of people that developed some
programming language?

On Mar 12, 6:48 pm, Steven <steven...@gmail.com> wrote:

[Jonathan Amsterdam]

On Mar 13, 5:07 am, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
> May I ask you a question: Have you ever developed your own programming
> language, or participated in a group of people that developed some
> programming language?

How is that relevant? Steven's points are perfectly cogent, whether he
is a language designer, a mere programmer, or a talking duck.

I think you are reading too much into what is an informal (in a
technical sense) spec. The spec is reasonably precise for a natural-
language document, but leaves much to general computer science common
sense. For example, the sentence "A variable declaration creates a
variable [and] binds an identifier to it" is unlikely to help you
unless you already know what "variable" and "binds" might mean. E.g. I
don't think the fact that bindings cannot be modified, only shadowed,
is ever mentioned. Similarly, you can't put too much pressure on words
like "shorthand" or "deduce". Absent an explicit statement about
rewriting, or perhaps the term "syntactic sugar," you can't infer that
the semantics of short declarations are completely determined by an
equivalent bit of source code.

So it would be a great help to all of us if you would produce a formal
spec. In so doing I'm sure about a hundred questions would come up,
which the Go designers and community would be delighted to clarify.

-Jonathan