Knowing from documentation whether an interface is holding a pointer or a struct?

[Joshua]

Hello,

My question is general, but for ease of communicating I'll use the specific example I ran into.

I'm very new and for my first project I'm working with the bleve library [https://pkg.go.dev/github.com/blevesearch/bleve].

One function I need, "Open", returns an interface, "Index".

I'd like to write my own function to act on this interface, and given that I have no idea what the dynamic value of the interface is, my first instinct is to rather pass a pointer to the returned interface into my function.

However, I see lots of calls of "If you're using pointers to interfaces a lot, you probably don't understand them".

Well, what am I not understanding?

My worry is that I have no idea what dynamic type is lurking within the interface, if it's a pointer to a struct, then I obviously don't mind passing it into my function.

However if it is in fact a humungous 1GB struct, then I really really don't want to be copying that around willy-nilly.

Is there a way in general to avoid this, without looking at the library source code to see what the actual concrete type is?

[Ian Lance Taylor]

In the current implementations a value of interface type is always a
pair of pointers. Even if the value of interface type happens to
refer to a 1GB struct, copying the interface value, including passing
it to a function or returning it from a function, always just copies
two pointers.

Ian

[Axel Wagner]

I would add that because the dynamic type of an interface value is not known at compile time, a variable of interface type really can't (in general) have a specific size.

If a function has an interface parameter, it must be possible to pass a value of *any* size to it. So even aside from what the current implementation does - any Go compiler must, in general¹, consider interfaces to be pretty-much-pointers.

"in general" because a compiler can, of course, determine that in a certain scenario the value doesn't have to be packed and pass it as-is. This is an optimization sometimes called "devirtualization". But in the general case, a compiler can't prove that (e.g. the dynamic value in an interface could be determined by a random number generator), so it will always be an optimization and the default always has to be a form of boxing into a constantly sized shape.

All of this is a good indication, from first principles, that you don't have to worry about the size of the dynamic value when passing it.

What's more, in general you should trust the author of the package you are using to give you a reasonable implementation of an interface. You shouldn't worry what the dynamic type and value in an interface is, unless you have very good reason to care. In this case, unless you notice that your code is very slow if you don't use a pointer (that would be "a very good reason to care"), you shouldn't optimize it. And if you notice, you should open a bug against that package :) Though as established, you won't.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUuv_qrrG8%3DdCQZv0%2BrKbnbW60XdOCwjp8M3EdOCxCNkw%40mail.gmail.com.

[Amnon]

I find that people coming to Go from C++ tend to use pointers everywhere so as to avoid copying of structs.
Once they get a bit more experience, they tend to use fewer pointers, and are happier to pass structs around.
Removing the "make everything a pointer" optimisation makes the code simpler, and often actually makes it run faster
as fewer values escape the heap. Allocation tends to dominate Go runtime, so it is worth doing a bit more
copying in order to get a bit less allocations. 

[Joshua]

Thanks all for the insights, I think a key takeaway for me is "Don't worry about it unless it's a problem", but it's also good to know that it (probably) isn't a problem!

I'm glad at least the semantics are the same, and I guess I'll cross the performance bridge if I ever come to it and someone tries to compile my code with an alternative/older compiler.

The FAQ [https://golang.org/doc/faq#pass_by_value] that raised this question for me still seems to be technically correct, but I will say the text definitely gives off a "If you're coming from C, pass big interfaces as pointers" vibe:

"Map and slice values behave like pointers: they are descriptors that contain pointers to the underlying map or slice data. Copying a map or slice value doesn't copy the data it points to. Copying an interface value makes a copy of the thing stored in the interface value. If the interface value holds a struct, copying the interface value makes a copy of the struct. If the interface value holds a pointer, copying the interface value makes a copy of the pointer, but again not the data it points to."

I wouldn't be surprised if other people from C/C++ fall into this trap, is there any chance the FAQ could be updated

[Rob Pike]

Can you explain the trap? I don't pick up that vibe, but I may be the author of that paragraph. Plus there is no such thing as a big interface. In the current implementation, all interfaces are the same size - a pair of words. You may still have a misapprehension.

Try the first half of this article I know I wrote: https://blog.golang.org/laws-of-reflection.

I recently translated a substantial C library into Go, and watching all the pointers disappear, at least syntactically (there were still slices), was marvelous.

-rob

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a891bbf5-9426-49b3-89c6-f185fe047b5en%40googlegroups.com.

[Axel Wagner]

TBH from that FAQ answer I would have come to the same conclusion as OP.

It literally says "Copying an interface value makes a copy of the thing stored in the interface value". But it doesn't. Assigning to an interface variable makes a copy of the value. Calling one of the methods on the interface makes a copy (to pass as the receiver). But copying the interface value itself doesn't.

And crucially, the difference means you are incentivized to use pointers to interfaces - because that way, you avoid copying the interface-value. In reality, of course, copying the interface value is harmless. And the parts *where* the copies happen, you have no control over as a user of the library - the initial boxing into an interface happens in the library, so does the decision if a method has pointer- or value-receiver.

ISTM the FAQ-answer only strays from being wrong by then adding "Actual implementations may apply optimizations to avoid copying as long as the optimizations do not change the semantics". But I do think at that point, the wrong impression already stuck. The FAQ answer is technically correct, but it should be more important what people take away from it.

I don't really know how to fix it, except removing the mentions of interfaces from that paragraph altogether. Because the most clear way to describe what's happening is to describe the interface as "a struct containing two pointers" and I'm generally opposed to using implementation-details to describe how the language works (plus, that description isn't even correct for all implementations). But maybe, by just not mentioning interfaces specifically *here*, but still describe that "copying a value" can mean "copying a pointer", if the value contains a pointer, the right impression ends up sticking.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOXNBZSKudZatL9tjO%2BPSxKycp-4hEBwXDvO9z0bQtrCLSsk6w%40mail.gmail.com.

[Brian Candler]

When you assign a regular (non-pointer) value to an interface variable, it does take a copy of that value:

https://play.golang.org/p/XyBREDL4BGw

Compare with what happens when the interface contains a pointer:

https://play.golang.org/p/UpZnHS0xDU1

As to whether the value is copied when you copy a non-pointer interface value to another interface variable, I'm having a hard time finding any way to demonstrate it one way or the other.  If you write a mutator method then it needs to take a pointer (which a non-pointer value doesn't satisfy).  And I can't find a simple way to get a pointer to the struct itself or its member, when it's held inside an interface value.  Maybe it's possible via reflect?

But in any case, I think the summary is:

* it's fine (and often a good idea) for an interface to contain a pointer value

* it's almost always wrong to take a pointer to an interface

[Joshua]

I can only give the opinion of someone who's new to the language, but maybe that is helpful for me to describe the learning journey a bit.

I'd seen references (hehe) to maps and slices "acting like pointers" and thus weren't costly to be passing around directly into functions, and so I went to look for some official clarification.

I found that FAQ, which seemed to answer my question exactly: Maps and slices act like pointers, copying them doesn't copy the data they point to. Interfaces do not, if you copy them you'll be copying whatever is "inside" them.

Maybe I read it that way because I went into it with that notion, but to me I think the most common case of someone reading this question is because they're asking themselves "So what exactly is expensive to copy?", to which the question replies "Concrete values and interfaces can be expensive, pointers/maps/slices aren't"

But as well, I don't know how to improve it, what it says is true, and the fact that in the current implementation the interface is always a pair of words is just that, an implementation detail.

And I'm probably overthinking it, but I in general try not to rely in implementation details when I write code, but perhaps that is not the Go mindset that I should be taking.

If in some C implementations you could pass a struct into a function without copying it (say, if the compiler could verify you were only reading it) , I would still indirect it with a pointer to account for the other compilers where such an operation is expensive.

Is it common in Go to target the "current implementation" when reasoning about the performance of a program?

[Axel Wagner]

On Sun, Jun 6, 2021 at 12:17 PM Brian Candler <b.ca...@pobox.com> wrote:

When you assign a regular (non-pointer) value to an interface variable, it does take a copy of that value:

https://play.golang.org/p/XyBREDL4BGw

Yupp, as I said :)

 

As to whether the value is copied when you copy a non-pointer interface value to another interface variable, I'm having a hard time finding any way to demonstrate it one way or the other.  If you write a mutator method then it needs to take a pointer (which a non-pointer value doesn't satisfy).  And I can't find a simple way to get a pointer to the struct itself or its member, when it's held inside an interface value.  Maybe it's possible via reflect?

If you know about the implementation, you can use unsafe: https://play.golang.org/p/IgDqfJ-DaDF

Given that the contained pointer doesn't change, no copy is happening. But of course, that assumes an implementation-view and the FAQ does point out that a copy doesn't *have* to happen.

Purely from a language perspective, there is no way to tell. Because, again, the FAQ entry is not *wrong*. Semantically, any implementation must behave *as if* a copy happens.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3e3f8b10-3806-4959-9f44-8fe484f67a24n%40googlegroups.com.

[Dan Kortschak]

On Sun, 2021-06-06 at 03:17 -0700, Brian Candler wrote:
> When you assign a regular (non-pointer) value to an interface
> variable, it does take a copy of that value:
> https://play.golang.org/p/XyBREDL4BGw

It depends on whether it's safe to leave uncopied or not. You can see
this here https://play.golang.org/p/q1729cX09BQ

[Robert Engels]

For me this is the most inconsistent and obtuse aspect of the Go language. It seems it would always be saner to treat interfaces as pointers. Which would mean if they had non pointer receiver methods might force more objects to be allocated on the heap - but it would prevent a lot of misunderstanding and subtle bugs.

> On Jun 6, 2021, at 5:43 AM, 'Dan Kortschak' via golang-nuts <golan...@googlegroups.com> wrote:

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e77e2a94d0c16514e74340a891f02e118e185d59.camel%40kortschak.io.

[robert engels]

For example, the fact that this code is broken is not intuitively obvious for any reader. It requires way too much scrutiny IMO.

https://play.golang.org/p/-f73t_Pm7ur

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/BBC4CAE4-9A6E-46EB-9E1D-681C40C5E8EF%40ix.netcom.com.

[jake...@gmail.com]

On Sunday, June 6, 2021 at 9:33:31 AM UTC-4 ren...@ix.netcom.com wrote:

For example, the fact that this code is broken is not intuitively obvious for any reader. It requires way too much scrutiny IMO.

https://play.golang.org/p/-f73t_Pm7ur

I would like to note that your example goes against the general advice that all methods should be on either pointers or values. Mixing value and pointer methods for the same types is a code smell. The code you posted is a good example of one of the reasons why.

The second to last paragraph in the FAQ section https://golang.org/doc/faq#methods_on_values_or_pointers says:

"If some of the methods of the type must have pointer receivers, the rest should too, so the method set is consistent regardless of how the type is used."

In your example, if MyEventRecorder.Log() is changed to have a pointer receiver, then the code works as expected: https://play.golang.org/p/MG10opC6Ect

[Gregg Townsend]

On Saturday, June 5, 2021 at 2:15:27 PM UTC-7 Joshua wrote:

However, I see lots of calls of "If you're using pointers to interfaces a lot, you probably don't understand them".

Well, what am I not understanding?

To answer this particular point:  In most cases (or most cases in actual practice?) an interface can be thought of as a pointer, and treated as such, so there's no reason to add the extra complication of another level of indirection.

I have to agree with the OP and Axel (and possibly others) that the subsequent prose about copying a struct as a consequence of copying an interface is confusing.  It surprised me to read that.  I like Brian's restatement that *assignment to an interface value* causes the copy. 

[Joshua]

> In most cases (or most cases in actual practice?) an interface can be thought of as a pointer, 

This is however, an implementation detail specific to the compiler you use though, correct?

And similarly, the wording of the FAQ is "fine", given that it's talking about the behaviour as the specification describes it (which makes no claim as to how an interface should be represented internally as far as I can tell).

Or am I completely barking up the wrong tree here?

I was under the impression that there can be (and are) many implementations/compilers for Go, and that everything on golang.org would be talking about the behaviour as the specification describes it, rather than as a particular compiler implements it.

[Marvin Renich]

* 'Dan Kortschak' via golang-nuts <golan...@googlegroups.com> [210606 06:43]:

"Copying an interface value makes a copy of the thing stored in the
interface value. If the interface value holds a struct, copying the
interface value makes a copy of the struct."

I believe this is patently false. It is neither conceptually required
by the language, nor is it true in practice, irregardless of any
optimizations.

1. It is required (as pointed out earlier in this thread) that
_assignment to_ an interface from a concrete value requires (at
least conceptually) copying the value (obviously, if the value is a
pointer type, the pointer, not the referent is copied).

2. It is also required (conceptually) that invoking a method on the
interface value, where the method on the concrete value has a
non-pointer receiver, makes a copy of the value to be used as the
receiver.

3. Furthermore, the complement of the first point is that when using
type assertion to obtain the concrete value from an interface, the
value (pointer if it is of pointer type) must be copied.

These three points, along with the fact that you cannot directly access
the concrete value without type assertion, are sufficient to make it
safe to copy an interface without making a copy of the concrete value
held by the interface, regardless of the implementation.

The statement in question should be removed from the FAQ because it is
just plain wrong, as well as because it is confusing and leads new Go
programmers to adopt bad programming practices.

I would replace it, and the subsequent statement about an interface
holding a pointer type, with a statement that says something along the
lines of

"Assigning a concrete value to an interface requires copying the
value (if the value is a pointer type, only the pointer is copied),
but copying an interface value never requires making another copy of
the concrete value, even if it is not a pointer type."

As an optimization (I don't know what the current compiler does), using
type assertion from one interface type to another interface type does
not require making a copy of the concrete value. Also, type assertion
to a concrete type can use the value from the interface directly if the
compiler can prove that it is not modified and doesn't escape.

...Marvin

[Marvin Renich]

* Joshua <joshua.o...@gmail.com> [210606 16:52]:

> > In most cases (or most cases in actual practice?) an interface can be
> > thought of as a pointer,
>
> This is however, an implementation detail specific to the compiler you use
> though, correct?

Well, sort of, but really no. Whether the compiler wastefully makes
multiple copies of large concrete values when interfaces are copied (or
passed as arguments) is an implementation detail. It is never necessary
to do so to conform to the spec, I would not expect any compiler to do
so, and the compiler you get from golang.org does not (as evidenced by
https://play.golang.org/p/q1729cX09BQ posted by Dan Kortschak).

I strongly suspect that the compiler writers got this right by clearly
thinking it through and realizing that it was never necessary, not by
applying an optimization for certain specific cases.

> And similarly, the wording of the FAQ is "fine", given that it's talking
> about the behaviour as the specification describes it (which makes no claim
> as to how an interface should be represented internally as far as I can
> tell).

And no, it is not fine; it is completely false.

See my recent reply to another msg in this thread.

...Marvin

[ben...@gmail.com]

I recently translated a substantial C library into Go, and watching all the pointers disappear, at least syntactically (there were still slices), was marvelous.

Side point: Rob, is this open source? If so, I'd be interested to see the side-by-side comparison of the C vs Go code (others might find it useful or interesting too).

-Ben

[Robert Engels]



There is no good reason that proper behavior should be dependent on understanding best practices. It should help with readability not correctness. Seems to me the compiler or Go Vet should prohibit this - in my review of the stdlib and other projects I can’t see any reason why it doesn’t. 

On Jun 6, 2021, at 11:10 AM, jake...@gmail.com <jake...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a9b4a8b3-0b2f-4935-807e-1cbca03a3b20n%40googlegroups.com.

[Axel Wagner]

FWIW I do tend to mix value and pointer receivers occasionally.

Sometimes a type needs a pointer receiver in one method, but a different method doesn't and it's more convenient to have a copy available for temporary operations.

Usually when I implement flag.Var, I make `Set` use a pointer receiver and `String` use a value receiver - because why not? There is no pointer needed to stringify a value.

Sometimes I have a slice type which should generally be used and passed as a value, but for convenience, I add a `push`/`pop` method to it, with a pointer receiver.

Of course, all of these things *could* be accomplished by just using pointer-receivers everywhere. But by that same logic, why have value receivers anyway? You *could* always use a pointer.

I don't think it's a huge problem to mix the two. It's true that this means the pointer and the value type implement different interfaces - but I don't think that's a huge problem either. I *definitely* doubt that it's "the most inconsistent and obtuse aspect of the Go language". I'm not saying it *never* caused any problems for me, but it definitely isn't as frequent a source of bugs as the behavior of closures in loops, or slices unintentionally sharing elements after append… All of which I'm fine with as well.

But FTR, this discussion is definitely off-topic in this thread. And it's also moot in general: It's not something we can realistically change now.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7C376006-EFA0-44BA-A036-B768078E4AA4%40ix.netcom.com.

[Robert Engels]

I think that is my point. The methods in the code I shared have a proper receiver type based on the requirements of the methods. It only “breaks” in the context of the usage which isn’t at all obvious. 

So it seems to me that go lint should at least complain that a struct has mutating methods so all methods should take pointer receivers. 

On Jun 7, 2021, at 9:19 AM, 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGYRdadYFidvhCKnNqKDeqBSJLWBEU%3DuCJrxr1SbhPepQ%40mail.gmail.com.

[Axel Wagner]

On Mon, Jun 7, 2021 at 7:42 PM Robert Engels <ren...@ix.netcom.com> wrote:

I think that is my point. The methods in the code I shared have a proper receiver type based on the requirements of the methods.

No, they don't. The `Log` method requires a pointer receiver, as it accesses state that is supposed to be shared. Once you fix that, the compiler will quite correct complain that the value does not implement the interface in question.

Note that the code behaves the same even if no interfaces are involved: https://play.golang.org/p/WpIzYYLOKn-

So it seems to me that go lint should at least complain that a struct has mutating methods so all methods should take pointer receivers.

I disagree.

[Marvin Renich]

* 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> [210607 10:19]:

> FWIW I do tend to mix value and pointer receivers occasionally.
> Sometimes a type needs a pointer receiver in one method, but a different
> method doesn't and it's more convenient to have a copy available for
> temporary operations.

Axel, I believe you already understand this, but for others following
along who want to understand when it might be appropriate to do this, I
would like to point out that for a type T with method Foo that has a
value receiver and method Goo that has a pointer receiver, and interface
I with methods matching Foo and Goo, T does not satisfy I, but *T does.
This is just something to keep in mind when weighing the possibilities.

...Marvin

[Robert Engels]

I don’t think that represents the problem fairly. In the non interface case I know I can’t accept a copy so I would declare the method as taking a pointer to the struct. 

With interfaces this is lost - as the interface is implicitly a pointer - but whether it points to a copy or the original is unknown because the dev writing the interface declaration and function using the interface only knows they need something with a Log().  

It is the implicit copying for the interface which makes it obtuse. 

On Jun 7, 2021, at 12:57 PM, Axel Wagner <axel.wa...@googlemail.com> wrote:



[Axel Wagner]

On Mon, Jun 7, 2021 at 11:42 PM Robert Engels <ren...@ix.netcom.com> wrote:

I don’t think that represents the problem fairly. In the non interface case I know I can’t accept a copy so I would declare the method as taking a pointer to the struct.

How methods are declared should, in general, not be a matter of whether or not they are assigned to an interface, but to whether or not they need a pointer. Again: Your code is incorrect without interfaces. The problem doesn't happen when you put that value into an interface - it happens when you pass a copy of it and expect it to refer to the original. Interfaces are just one way to create such a copy, but they do not matter for the correctness of this code and for whether or not that method needs a pointer receiver (it does).

But again, to be clear: I'm not saying problems like this *never* happen and I'm not even saying that interfaces may obscure it in some cases. Just that a) the root cause here is that your method really needs to take a pointer-receiver, interfaces or not and b) that it seems very much an overstatement to me to call this "the most inconsistent and obtuse aspect of the Go language".

With interfaces this is lost - as the interface is implicitly a pointer

Well, it seems a bad idea to say that interfaces are implicitly pointers then. That seems to indicate that Rob's original phrasing is indeed an important clarification - the language behaves as if the value contained in them is copied when the interface value is copied.

It seems the confusion here is, that you assume it's not. And that interfaces act as a pointers, when they don't.

[Axel Wagner]

BTW, just to nail down the point of that code being wrong without interfaces: Your usage of `atomic` in `Log` is superfluous. You are operating on a local variable, so there is no possibility of concurrent modification. Your code is equivalent to this: https://play.golang.org/p/zYG0zTsk-2a

The only reason to use `atomic` here (and why you used it) is if that memory could be shared between goroutines. For that to happen, you need a pointer receiver though.

I refuse to believe that interfaces have anything to do with this obfuscation here. There is more than enough indication of it being wrong in any case.

[Robert Engels]

I think that is why it is inconsistent and obtuse to me.  The Log() method doesn’t need a pointer receiver. It works fine without it. It only needs a pointer receiver because when passed to a function declared as taking an interface a copy is made (and a reference to the copy held). This copy is implicit not explicit. This is a difficult model when working on large multi person projects - thus the “best practice” of using only pointer or value receivers I think. Suggesting that the linter flags code not adhering to this seems to have very little downside if any. 

A quick search on the web (and the faq language) makes me think I am not alone in this opinion.

I guess I still don’t see the downside to linter changes. As I said I reviewed a lot of the stdlib and I don’t see any obvious mixed receiver structs. They can be disabled on a per struct basis if needed. 

Anyway it was just my opinion based on my experiences. 

On Jun 7, 2021, at 6:05 PM, Axel Wagner <axel.wa...@googlemail.com> wrote:



[Robert Engels]

The pattern of a background stats collector is a common one. The atomic is required not optional. 

On Jun 7, 2021, at 6:16 PM, 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGVyvYYpQhCp_JkxN9EvgZ4FXJ8_WpxseJOB1OR7qt6ww%40mail.gmail.com.

[Axel Wagner]

On Tue, Jun 8, 2021 at 1:25 AM Robert Engels <ren...@ix.netcom.com> wrote:

I think that is why it is inconsistent and obtuse to me.  The Log() method doesn’t need a pointer receiver. It works fine without it.

I don't understand how you can continue to say that. I've linked the code several times. It does not work. Or, if it does work, it also works with interfaces. The behavior is the same in both cases.

 

It only needs a pointer receiver because when passed to a function declared as taking an interface a copy is made (and a reference to the copy held). This copy is implicit not explicit.

Yes. But (as the FAQ entry points out) copying values when passing is how Go works in general.

 

This is a difficult model when working on large multi person projects - thus the “best practice” of using only pointer or value receivers I think. Suggesting that the linter flags code not adhering to this seems to have very little downside if any.

It has the downside of flagging correct code as incorrect.

[Axel Wagner]

On Tue, Jun 8, 2021 at 1:26 AM Robert Engels <ren...@ix.netcom.com> wrote:

The pattern of a background stats collector is a common one. The atomic is required not optional. 

It might be a common pattern, but it uses a pointer-receiver in that case. The atomic operation is not required, it operates on a local variable. Again, I don't understand how you can make statements that are so clearly wrong.

Feel free to try running it in the race detector without an atomic operation. Feel free trying to get the race detector to trigger without the atomic access, but keeping it silent when you add it. You'll find that this needs a pointer receiver. Because otherwise the function is operating on a local variable.

[Robert Engels]

We agree. It needs a pointer receiver to work. The atomic is also needed in this case for background logging. 

The problem in this case is that recordEvents() has to document that the  EventLogger passed to recordEvents() must have a pointer receiver for the Log() method. There is nothing in the language that allows me to declare it nor the compiler to enforce it.

If you don’t see this as suboptimal and an area for improvement I am not sure what else I can say.

And by the way, linters often flag correct code - that is why they have disable options. They try to enforce the most common cases - and by the recommendation in the faq to only use receivers of the same type - it seems appropriate to me to have the linter flag this. 

As to this being in my opinion the most inconsistent and obtuse aspect of Go - that is my opinion. Curious, what do you think would take the top spot?

On Jun 7, 2021, at 6:34 PM, Axel Wagner <axel.wa...@googlemail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfHibUpDHQU9m8%3DrLYtnDj%3DFY01nkuP4k0Giow-hCbhNgQ%40mail.gmail.com.

[Axel Wagner]

On Tue, Jun 8, 2021 at 2:05 AM Robert Engels <ren...@ix.netcom.com> wrote:

We agree. It needs a pointer receiver to work. The atomic is also needed in this case for background logging. 

The problem in this case is that recordEvents() has to document that the  EventLogger passed to recordEvents() must have a pointer receiver for the Log() method. There is nothing in the language that allows me to declare it nor the compiler to enforce it.

It is possible to write a working implementation of that interface without a pointer receiver - it just needs to *contain* a pointer: https://play.golang.org/p/Xm6ASGcCyhR

You could also have a slice type, which also can do modifications without a pointer receiver. Or a map-type. Or a channel.

If you would restrict an interface to require pointer-receiver, you would wrongly restrict the implementer from all these possibilities.

As is the common wisdom, the user of an interface should not care what the concrete type implementing an interface is (except if it needs to do a type-assertions). It's the same wisdom that applies to people wanting to check if an interface contains a nil-pointer: That check relies on the assumption that the interface contains a pointer, which shouldn't be nil and that's not something that should concern the user of an interface.

Again, to be abundantly clear (you still seem unwilling to acknowledge this): The problem with your code is not the definition or usage of the interface. It's the definition of the method that is wrong. The interface-definition is fine and works fine.

If you don’t see this as suboptimal and an area for improvement I am not sure what else I can say.

I just want to state again, clearly, that all I objected to was you calling this "the most inconsistent and obtuse aspect of the Go language", which I perceived (and still do) to be an overstatement. "It is suboptimal" or "it is an area of improvement" are both significantly weaker statements, which I find less objectionable.

 

Personally, I still don't think it is a huge problem. And the fact that you where having a lot of trouble coming up with an example showing it to be one (the one you posted doesn't - again, it doesn't, in any way, change behavior when using or not using interfaces) is, in my view, a testament to that.

And by the way, linters often flag correct code - that is why they have disable options. They try to enforce the most common cases - and by the recommendation in the faq to only use receivers of the same type - it seems appropriate to me to have the linter flag this. 

I'm opposed to a linter flag, because it would flag correct code I regularly write. In general, linters should not be ignored - they either shouldn't be run, or they should be followed. Note that golint has no option to selectively disable a particular instance of a warning - the only way to silence a warning is to change the code. But I don't want to use a pointer receiver, if a value receiver is more appropriate.

If golint or go vet would start flagging this, I would likely follow the advice it's giving. Because that's how linters and static checks are supposed to be used - to enforce consistency. But I'd be sad doing it. Which is why I don't want them to flag it.

I'm less opposed to the FAQ entry. Simpy because an FAQ entry can be more easily ignored where it makes sense. If you will, it is one step in stringency below a linter. I'm fine defending my choice in a code review, but I don't want to defend it to a linter.

 

As to this being in my opinion the most inconsistent and obtuse aspect of Go - that is my opinion. Curious, what do you think would take the top spot?

I'm not sure. I don't like putting things in absolute order or claiming something is "the most X" for exactly that reason - it almost always turns out to be an overstatement.

Empirically, the issue of nil-pointers in interfaces not being nil seems to take one of the top spots, even though I don't fully understand why.

To me, concurrency in Go is extremely subtle and I would generally advice novices to stay away from it at first (or stay with extremely simple constructs), because they are likely to get it wrong.

Details of how Go handles constants and type-identity/assignabiity is what is probably most often tripping me, personally, up in questions/quizzes about Go. But it rarely comes up in practice.

The lack of co/contravariance is probably one of the things I miss the most from the language.

It really depends on what you're asking. And I'm very likely forgetting things while being put on the spot.

It's just a lot easier to make relative judgments, than absolute ones.

[Marvin Renich]

* 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> [210607 19:06]:

> Well, it seems a bad idea to say that interfaces are implicitly pointers
> then. That seems to indicate that Rob's original phrasing is indeed an
> important clarification - the language behaves as if the value contained in
> them is copied when the interface value is copied.

I agree with most of what you are saying to Robert Engels, but I
disagree with this. The FAQ needs to make it clear that making a copy
of the value happens when assigning a concrete value to an interface and
when extracting a concrete value from an interface, but that the
interface passed around does _not_ require the allocation overhead of
copying the concrete value every time the interface is copied (even if
it behaves, whenever the interface is used, as if a copy is made).

It is clear from this thread that this is a confusing topic, and also
that programmers new to Go are reading this FAQ entry and believing,
incorrectly, that to optimize memory allocations they should use
pointers when it is not necessary to do so.

In other words, there are two distinct aspects that both need to be
expounded in the FAQ: the external behavior and the performance-related
consequences. People are jumping to the wrong conclusion because the
FAQ says "copying the interface value makes a copy of the struct". This
is simply untrue, and not necessary according to the language spec.

The correct statement is that "copying the interface value behaves as if
the struct is copied without actually needing to make a new copy of the
struct". That second part is important.

...Marvin

[Robert Engels]

(I think you pasted the wrong link - that is my code). 

It is not about being unwilling to admit it. Your explanation/reasoning has not convinced me. 

Imagine some library declares the EventLogger interface as shown. Acceptable. Someone writes the RecordEvents() method taking an EventLogger. Acceptable. 

Now, I have a struct I want to use with as an EventLogger (badly named - really EventSource). The code I wrote works fine. Test cases (of Log()) work fine. It fails when used as a source to RecordEvents() (and similar held reference patterns). 

How do you protect against this? What is the mechanism as a library author?

Clearly this is a trivial example but similar patterns are everywhere. 

Compare the Go interface handling with Java’s everything is a reference - much simpler - and then adding value types that are explicit. Or a similar implementation in Rust. In both cases knowing you wrote a correct implementation is much easier. Java has since added annotations for aspects like “thread safe” that cover the atomic aspects. 

I like Go. A lot. I’ve designed and built systems with millions of LOC. Pointing out aspects that might benefit from changes should be encouraged - if not it’s a religion not a programming language. 

On Jun 7, 2021, at 7:40 PM, Axel Wagner <axel.wa...@googlemail.com> wrote:



[Robert Engels]

Sorry - correct link. I missed the subtle change.

On Jun 7, 2021, at 8:18 PM, Robert Engels <ren...@ix.netcom.com> wrote:



[Marvin Renich]

* Robert Engels <ren...@ix.netcom.com> [210607 21:18]:

> (I think you pasted the wrong link - that is my code).

subsequent correction acknowledged; assuming the following discussion is
about https://play.golang.org/p/-f73t_Pm7ur

> It is not about being unwilling to admit it. Your
> explanation/reasoning has not convinced me.
>
> Imagine some library declares the EventLogger interface as shown.
> Acceptable. Someone writes the RecordEvents() method taking an
> EventLogger. Acceptable.
>
> Now, I have a struct I want to use with as an EventLogger (badly named
> - really EventSource). The code I wrote works fine. Test cases (of
> Log()) work fine. It fails when used as a source to RecordEvents()
> (and similar held reference patterns).

I have to agree with Axel. This code is not fine, and it is easy to at
least notice that it is suspect, even if it requires more careful
examination to determine that it is wrong. The fact that the Log method
has a value receiver and the Inc method has a pointer receiver is a very
obvious clue that this code needs careful review. Next, Log takes a
value receiver, but is using synchronization (whether it is atomic or
mutex doesn't matter); this should be where the programmer realizes that
the code is broken. And, as Axel keeps saying, it has nothing to do
with interfaces; the method on the struct is broken without any
interface being involved.

You say that test cases of Log work fine, but they are only fine in a
non-concurrent environment. The instant you test Log (without
interfaces) in a concurrent program it fails in an obvious manner.

Interfaces, and their ability to potentially be satisfied by either a
non-pointer type or a pointer type, depending on the type, is not the
problem here.

...Marvin

[Axel Wagner]

On Tue, Jun 8, 2021 at 3:16 AM Robert Engels <ren...@ix.netcom.com> wrote:

Now, I have a struct I want to use with as an EventLogger (badly named - really EventSource). The code I wrote works fine. Test cases (of Log()) work fine.

This is where the refusal to acknowledge comes in. It does not work fine. I don't understand how you can look at the code without interfaces, run it, see that it behaves *exactly* the same as your code with interfaces and still make a statement that this code works fine. This isn't a matter of being convincing. I'm not making an argument here. The code you wrote to show that the method is broken shows that it's broken if you use interfaces or not. How can you look at that and deny that the code executes the way it does, when you can actually run it for yourself?

 

If you wrote tests for those methods and those tests passed, then clearly your tests are insufficient. Because you, personally, have posted a test case in this thread, which shows that the `Log` method does *not* work fine. Without any interfaces.

That's what baffles me. How you can hold on to the narrative that the code without interfaces works fine and the code with interfaces breaks, when that's so easily verifiable and shown to be false, just by clicking "Run" on the playground.

Even if you would try to make the argument (but, again, you are simply refusing to even acknowledge, you aren't even making a counter argument) that the tests without interfaces would use a pointer (but that's changing more than just whether or not you are using interfaces - and if you do the same change using interfaces you again get the same execution trace), all that's needed to still see that's broken is to run it using `-race`.

There is no opinion here. These are easily verifiable facts. Just repeating "the code works fine" doesn't change the fact that people can run it and see that it's not.

[Axel Wagner]

On Tue, Jun 8, 2021 at 3:16 AM Robert Engels <ren...@ix.netcom.com> wrote:

I like Go. A lot. I’ve designed and built systems with millions of LOC. Pointing out aspects that might benefit from changes should be encouraged - if not it’s a religion not a programming language. 

FTR, this is the second time you are making that twist and I already called out the first and I don't appreciate that you are repeating it.

"This aspect might benefit from a change" is a significantly weaker statement that using a superlative like "the most inconsistent and obtuse aspect of the Go language".

[Axel Wagner]

On Tue, Jun 8, 2021 at 6:36 AM Marvin Renich <mr...@renich.org> wrote:

You say that test cases of Log work fine, but they are only fine in a
non-concurrent environment.  The instant you test Log (without
interfaces) in a concurrent program it fails in an obvious manner.

nit: I don't think concurrency has anything to do with it either. The failure mode is making a copy and expecting the copy and the original to share memory. If anything, concurrency (in a test) would make it more likely to get hidden, by increasing the likelihood that a closure is used, which implicitly shares a pointer: https://play.golang.org/p/Gwj9GScjQBJ

Of course, concurrency then also makes the failure easy to see, as long as you remember to run your tests with `-race`.

 

FWIW I agree with Robert that it's relatively easy to write a test for this that never copies the value (though even then, if you think about using atomics you definitely should think about writing a concurrent test and running it with `-race` enabled, which should show the problem).

I disagree with him, however, that interfaces make it more likely to run into the problem when *using* the code. Any even remotely realistic usage of that code is broken. Even if you failed to write tests which surface that breakage.

Interfaces, and their ability to potentially be satisfied by either a
non-pointer type or a pointer type, depending on the type, is not the
problem here.

...Marvin

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/20210608043609.gguu3t3hqbsziema%40basil.wdw.

[robert engels]

OK, I will try one more time.

The following code is works fine from the developers perspective:

https://play.golang.org/p/gC1XsSLvovM

The developer says, oh cool, I see this great new 3P library that does background logging - I want to use that instead. Hey, I already implement the EventLogger interface, so no problem, I take out my manual logging code, and make a call to recordEvents(EventLogger).

Hmm, my program isn’t logging properly any more. Oh, it says background logging - that likely means concurrency - so maybe I have a race condition, so I run it under —race. Hmmm, says it's fine and it still isn’t working. :(

Eventually the developer figures out that the call to recordEvents() is making a copy, and so needs pointer receiver and to create a pointer based interface reference (and add the atomic calls). It’s not clear to me how the library author would document things to avoid this scenario.

If you don’t see that the above is suboptimal and an AFI I am not sure what else I can say. That a person of your caliber writes code that differs from the recommendation in the FAQ (the mixing of receiver types) is a flag to me that one of the approaches is probably not correct.

‘go vet’ already has the ability to disable certain checks. Adding a check to ‘go vet’ to detect mixed receiver types (which the FAQ says is not recommended) seems reasonable and will make life easier for many beginner Go programmers - and some seasoned ones as well :)

The duplicitous and transparent nature of pointer/value receivers and interfaces is a source of confusion. I think being explicit would have been a better choice here but that horse has left the barn.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGa5B99OO8oTxJOWFSuJPinPF3b1QJWEuqMD4ZPAiCi%3DQ%40mail.gmail.com.

[Marvin Renich]

* 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> [210608 02:54]:

> On Tue, Jun 8, 2021 at 6:36 AM Marvin Renich <mr...@renich.org> wrote:
>
> > You say that test cases of Log work fine, but they are only fine in a
> > non-concurrent environment. The instant you test Log (without
> > interfaces) in a concurrent program it fails in an obvious manner.
> >
>
> nit: I don't think concurrency has anything to do with it either. The
> failure mode is making a copy and expecting the copy and the original to
> share memory.

I agree.

> FWIW I agree with Robert that it's relatively easy to write a test for this
> that never copies the value (though even then, if you think about using
> atomics you definitely should think about writing a concurrent test and
> running it with `-race` enabled, which should show the problem).

Sure.

> I disagree with him, however, that interfaces make it more likely to run
> into the problem when *using* the code. Any even remotely realistic usage
> of that code is broken. Even if you failed to write tests which surface
> that breakage.

Right. This is just one of many aspects of programming that require
careful consideration to get right. As you keep saying, "this is
suboptimal" applies, but "the most inconsistent and obtuse aspect of the
Go language" is way too extreme of an assertion.

...Marvin

[Dan Kortschak]

An error like this would be found in any reasonable set of tests for
the type. Minimally, if the type is intended to be passed to functions
in order to be able to be used, a test would involve that and that
would immediately surface the error.

What you seem to be asking for is a way for the language to enforce
contracts on behaviour from the source and you seem to put that at the
feet of interface handling. However, it is clearly something that
arises without needing interfaces in the language; a user of the
MyEventRecorder type would already find problems if they pass the value
to a function without needing to do that in an interface value. Without
significant addition of annotations for implementations it is not
possible for the language to understand the contractual obligations of
an implementation, and these kinds of annotations are in opposition to
the design of the language. Other languages do have these kinds of
marks, but they are not Go (thankfully).

In terms of documentation of expectations of an interface if that were
the situation, there are examples in the standard library where it's
described that implementations must be able to mutate themselves — be a
pointer receiver (or other reference-like type).

[Robert Engels]

I think the playground code I supplied is essentially a test case - and it works in the absence of concurrency or other held references (like putting it in a map).

I guess the bottom line for me is that faq says do not mix receiver types. There is either a valid reason for this or it should be removed. I think it is the former as it makes more usage patterns resilient to hard to track bugs.

I must be very dense but I still don’t see the rationale behind not stating : use value receivers for immutable types and pointer receivers otherwise. There may be some fringe cases where this doesn’t hold - but if you can write the entire stdlib with this pattern it seems fairly stable.

> On Jun 8, 2021, at 5:03 PM, 'Dan Kortschak' via golang-nuts <golan...@googlegroups.com> wrote:

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/cce7af413d7cb56b4fd653703f1d49d6f5f7e7c6.camel%40kortschak.io.

[Marvin Renich]

* Robert Engels <ren...@ix.netcom.com> [210608 18:30]:

> I think the playground code I supplied is essentially a test case -
> and it works in the absence of concurrency or other held references
> (like putting it in a map).

But it is one simple test case, not an appropriate set of tests. Saying
that code passes one test says very little about its correctness. It is
often very easy to intentionally (or inadvertently) write a single test
that passes, even though the code being tested is broken.

> I guess the bottom line for me is that faq says do not mix receiver
> types. There is either a valid reason for this or it should be
> removed. I think it is the former as it makes more usage patterns
> resilient to hard to track bugs.

I think your MyEventRecorder is good anecdotal evidence that it is
usually better to not mix receiver types. I don't think anyone is
saying that the FAQ is wrong, just that it is not an absolute. I don't
think Axel was suggesting that this should be removed, he was just
giving an aside that he sometimes does not follow that advice, but he
knows why he does and accepts the responsibility for any mistakes as a
result.

> I must be very dense but I still don’t see the rationale behind not
> stating : use value receivers for immutable types and pointer
> receivers otherwise. There may be some fringe cases where this doesn’t
> hold - but if you can write the entire stdlib with this pattern it
> seems fairly stable.

The way I read it, that is the primary advice in the
https://golang.org/doc/faq#methods_on_values_or_pointers section. It
also gives some other considerations that may mitigate that advice.

...Marvin