"for ... range" over array of structs copies values

[bolson]

not news to the people who have better internalized Go, but this bit
me today:

type Foo struct {
a, b int
}
foos := make([]Foo, 10)
foops := make([]*Foo, 10)
for i, val := range foos {
foops[i] = &val // This is doomed to failure. Actually everything
in foops gets a pointer to the one local instance of Foo that is the
local loop variable 'val' that gets assigned into.
}

If it's not already in the tutorials or specification somewhere, I
think it would be worth mentioning this consequence of the "for ...
range" semantics on an array of struct. Actually after figuring out
what was going on I looked it up in the release 60.3 specification and
it does say basically "range is assignment", so, struct-value-copying.
Mostly surprising because it's different than what I wanted and
expected from using other languages (C++, Java, Python). I guess it's
most like good-old-C, either explicit pointer or value, no automatic
or implicit or magic references.

[Jesse McNelis]

On 25/12/11 09:58, bolson wrote:
> not news to the people who have better internalized Go, but this bit
> me today:
>
> type Foo struct {
> a, b int
> }
> foos := make([]Foo, 10)
> foops := make([]*Foo, 10)
> for i, val := range foos {

> foops[i] =&val // This is doomed to failure. Actually everything

> in foops gets a pointer to the one local instance of Foo that is the
> local loop variable 'val' that gets assigned into.
> }
>
> If it's not already in the tutorials or specification somewhere, I
> think it would be worth mentioning this consequence of the "for ...
> range" semantics on an array of struct. Actually after figuring out
> what was going on I looked it up in the release 60.3 specification and
> it does say basically "range is assignment", so, struct-value-copying.
> Mostly surprising because it's different than what I wanted and
> expected from using other languages (C++, Java, Python). I guess it's
> most like good-old-C, either explicit pointer or value, no automatic
> or implicit or magic references.

Range in Go works pretty much exactly the same as 'for in' in python and
the similar 'for each' construct in Java or the same construct in pretty
much every other language that has it.

If you could take the address of things in python or Java you'd notice
the same thing. It's the only efficient way to do such a construct and
it's pretty easy to understand.

- jessta

[bolson]

Before going further, **my point here is that in learning Go, I got
tripped up by behavior that wasn't what I expected**, it may in fact
be good and reasonable and consistent with other principles, but maybe
we can do more to help people understand and make those things clear.

Except that in Python and Java all Objects are reference variables.
// Java:
for (MyClass x : arrayOfMyClass) {
x.foo = bar; // fine
}

// Go:
var arrayOfMyStruct []MyStruct
for i, x := range arrayOfMyStruct {
x.foo = bar // doom. useless assignment to local variable x.
}

Python and Java don't even really have the ability to make an array of
struct. They can only do array of pointer to struct. I'm glad Go can
do a real array of struct for memory efficiency reasons.
I think maybe I expected the range variable to be something like a C++
reference type (which yes, of course now I realize Go doesn't have,
only pointers and values).
Maybe I want somewhere it to be explicitly stated in a helpful way:
for i, val := range someArray {
}
// Is equivalent to
for i = 0; i < len(someArray); i++ {
val := someArray[i] // note that assignment is by value is not a
reference
}

I thought: "val is the element in the array". Simple. But not correct
in this case; not in the way I wanted to use it.
Isn't that a natural thing to think about a foreach loop? "for each
element in the array", not "for a copy of each element in the array"?

Maybe Go needs a variation on the range operator that takes array []T
and yields (int, *T) ? (call it 'rangep' or 'prange' or 'rangeref' or
something)


And a meta note. I got one response here and one personally basically
saying that my thinking is baseless and stupid and foolish. If too
many people trying to learn the language get that, fewer people will
try to learn the language. Good documentation and good community (and
yes, good technology) will build Go. :-/

[Nigel Tao]

On 25 December 2011 11:21, bolson <brian...@gmail.com> wrote:
> Maybe Go needs a variation on the range operator that takes array []T
> and yields (int, *T) ? (call it 'rangep' or 'prange' or 'rangeref' or
> something)

for i := range a {
a[i].foo = bar
}

For example:
----
package main
func main() {
a := make([]int, 4)
for i, v := range a {
println(&v, &a[i])
}
}
----
prints
----
0x7fee85916f44 0xf8400020f0
0x7fee85916f44 0xf8400020f4
0x7fee85916f44 0xf8400020f8
0x7fee85916f44 0xf8400020fc
----

Oh, and a newbie nit on Go terminology: []int is a slice, [4]int is an array.

[Nigel Tao]

On 25 December 2011 11:21, bolson <brian...@gmail.com> wrote:

> And a meta note. I got one response here and one personally basically
> saying that my thinking is baseless and stupid and foolish. If too
> many people trying to learn the language get that, fewer people will
> try to learn the language. Good documentation and good community (and
> yes, good technology) will build Go. :-/

You got only one response in two hours because it's the holiday season
in many parts of the world, and because it's a mailing list, not IRC.
And I don't read Jessta's response as saying any of baseless, stupid
or foolish.

The point is simply that Go structs behave like C structs and not like
Java objects, for things like assignment. If your prior experience is
C, then it is perfectly intuitive. If your prior experience is Java,
then some things will simply be different.

It would be great to have "Go for experienced Java programmers" or "Go
for experienced Python programmers" documentation, but I don't have
time to write it. I was surprised to find out that a[i:] and a[:j]
mean similar things in Python and Go, but a[:] means something totally
different. As a Go programmer, I found Python's behavior surprising.
I'm sure that an experienced Python programmer would have found Go's
behavior surprising. But I wouldn't call either myself or my
Python-loving friend stupid. We simply come from different
backgrounds.

[Dmitry Vyukov]

On Sun, Dec 25, 2011 at 3:21 AM, bolson <brian...@gmail.com> wrote:

Before going further, **my point here is that in learning Go, I got
tripped up by behavior that wasn't what I expected**, it may in fact
be good and reasonable and consistent with other principles, but maybe
we can do more to help people understand and make those things clear.

Except that in Python and Java all Objects are reference variables.
// Java:
for (MyClass x : arrayOfMyClass) {
 x.foo = bar; // fine
}

// Go:
var arrayOfMyStruct []MyStruct
for i, x := range arrayOfMyStruct {
 x.foo = bar // doom. useless assignment to local variable x.
}

Is it really relevant whether it's a slice of structs or ints? Go seems to be consistent in this respect.

[i3dmaster]

I think that everything in Go is pass by value (same as Java and Python). The difference is that primitive types in Java are passed by value explicitly where objects are passed by value of its pointer (Java made it impossible to pass a value of the object itself). But still, it is pass by value (the value of the pointer). I think in this respect, Go is consistent. These languages do not have the c++ "reference" concept. 

[Vadik Vygonets]

On Dec 25, 1:21 am, bolson <brian.ol...@gmail.com> wrote:
> Maybe I want somewhere it to be explicitly stated in a helpful way:
> for i, val := range someArray {}
>
> // Is equivalent to
> for i = 0; i < len(someArray); i++ {
>   val := someArray[i] // note that assignment is by value is not a
> reference
>
> }

Except that it's not. In the first case, it's the same val in all
iterations of the loop; in the second case, it's a new val in every
iteration. Now, why would you care whether the values sit in the same
variable or not, I hear you asking? Suppose that from within a loop
you want to call a function that accepts a niladic function and starts
it in a new goroutine, like this one:

func start(f func()) {
// possibly do something, and eventually
go f()
}

You can't pass arguments to it, you have to make a closure, like so:

func main() {
a := []int{0, 1, 2}
for _, v := range a {
start(func(){fmt.Printf("%d\n", v)})
}
time.Sleep(time.Second)
}

If you compile it, you'll see that it prints "2" three times. But if
you change the loop to:

for i := range a {

v := a[i]
start(func(){fmt.Printf("%d\n", v)})
}

you'll see the result you probably would expect. Isn't programming
fun?

Vadik.

[Gustavo Niemeyer]

(...)

> I thought: "val is the element in the array". Simple. But not correct
> in this case; not in the way I wanted to use it.
> Isn't that a natural thing to think about a foreach loop? "for each
> element in the array", not "for a copy of each element in the array"?

The confusion comes from misunderstanding that aspect. What you get as
a value of iterations, parameters of functions, etc, is consistently a
_copy_ in most languages [1]. The question is just what is being
copied, not whether it is being copied or not. E.g.:

In Go:

for _, x := range []int{1,2,3} {} // x is a copy of the integer
for _, x := range []*int{a, b, c} {} // x is a copy of the pointer
to the integer value
for _, x := range []*T{d, e, f} {} // x is a copy of the pointer
to the T value
for _, x := range []T{g, h, i} {} // x is a copy of the T value

In Python:

for x in [1, 2, 3]: ... // x is a copy of the pointer (!) to the
integer object
for x in [a, b, c]: ... // x is a copy of the pointer (!) to each
object in the list

It's easy to miss the fact that e.g. Python works with pointers all
the time because we don't see the star there, but it's a fact: for
good and for bad, you're dealing with pointers all the time, even for
simple things such as integers [2]:

>>> a, b = int(1e9), int(1e9)
>>> object.__repr__(a); object.__repr__(b)
'<int object at 0xb47570>'
'<int object at 0xb475b8>'
>>> for i in [a, b]: object.__repr__(i)
...
'<int object at 0xb47570>'
'<int object at 0xb475b8>'

[1] Reference parameters in C++ being one of the notable and ugly
exceptions. http://j.mp/tNtKGQ
[2] PyPy and psyco are able to JIT-optimize them, though

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/plus
http://niemeyer.net/twitter
http://niemeyer.net/blog

-- I'm not absolutely sure of anything.

[bolson]

If I were making a Go style-guide for a group which made it's product
in Go, I'd have to include something like this:

`for iter, val := range X {}` shall only be used if val is a bool/
number/pointer/string/slice/map/interface/func type assigned from an
array/slice/map of such. val MAY NOT be a struct or array type from an
array/slice/map of such. `for iter := range X {}` may be used for any
type.

I think what I'm going for here is 'no copying anything bigger than a
CPU register' (except maybe an int64 on a 32 bit cpu, oh well, I guess
virtual registers count too).

And then I'd want some static checking to emit warnings when this was
violated.

[Jan Mercl]

There are completely legitimate and sane uses for 'for i, v := range sliceOfStruct { ... }', both the guide 'rule' and the compiler warning would be outright wrong.

[Pierre Gaston]

It's still very much a gotcha if you come from other languages, I
think that stressing that the range operator is not just an
alternative way to write the "naive" loop on the index is well worth
half a sentence in the specs (or elsewhere).(eg, "as a result the
element of the array is a copied in the variable even if it's a
struct")
Aside from that I wonder if there is a way to loop on a map that
doesn't involve copying the elements of the map.

[Pierre Gaston]

hmm I guess using _ for the element can do the trick

[Michael Jones]

I've asked for this too. My proposal is this:

    for index, value, addressOfValue := range thing {

    }

which is a compatible addition to the existing syntax. typical use would be:

    for _, _, p := range stuff {

        p.fieldName = 7

    }

On Tue, Sep 25, 2012 at 5:13 PM, <dbea...@google.com> wrote:

What would be great is if I could do something like:

    for _, &foo := range foos { ... }

and get a pointer instead, but I haven't seen a way to do this (I could be missing something).

--

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Jan Mercl]

On Tue, Sep 25, 2012 at 1:43 PM, <dbea...@google.com> wrote:
> Now everything breaks!

The only thing which should happen is that tests don't pass anymore.

-j

[David Beaumont]

On 25 September 2012 14:32, Michael Jones <m...@google.com> wrote:
> I've asked for this too. My proposal is this:
>
> for index, value, addressOfValue := range thing {
> }
>
> which is a compatible addition to the existing syntax. typical use would be:
>
> for _, _, p := range stuff {
> p.fieldName = 7
> }

I'd prefer:

for _, &p := range stuff {
p.fieldName = 7
}

which I think is unambiguous syntactically.

David

--
David Beaumont :: Îñţérñåţîöñåļîžåţîờñ Libraries :: Google
Google Switzerland GmbH., Brandschenkestrasse 110, CH-8002, Zürich - Switzerland
Tel +41 44 668 1800 :: Fax +41 44 668 1818

[Alexei Sholik]

I also support the &p syntax as it makes the intention perfectly clear. Using

    i, obj, address := for range {

    }

is redundant. We can get object's address ourselves. And using _ does not necessarily mean, that object will not be copied, it may simply be discarded afterwards.

--

--
Best regards
Alexei Sholik

[Ethan Burns]

On Tuesday, September 25, 2012 7:43:28 AM UTC-4, (unknown) wrote:

Consider:

type Foo struct {

  bar *Bar

} 

type Bar struct {

  value int

}

You do:

  foos []Foo := ...

  for _, foo := range foos {

    foo.bar.value = nextValue();

  }

Now this work just fine & dandy until someone decides that Bar (being a small struct) can be inlined in Foo.

type Foo struct {

  bar Bar

} 

Now everything breaks! Remember that this code change will not require anyone to revisit the code in the for loop, which could be in another class/package altogether (in C++ you'd at least have to turn a '->' into a '.' to get it to compile again).

What would be great is if I could do something like:

    for _, &foo := range foos { ... }

and get a pointer instead, but I haven't seen a way to do this (I could be missing something).

Adding & to the range clause syntax just saves a single line of code:

for i := range foos {

    foo := &foos[i]

    ...

}

The above seems fine to me, and it doesn't require adding anything more to the language. I also believe that the compilers are smart enough to elide the bounds check in simple cases like this, if you are worried about that.

Best,

Ethan

 

[Alexei Sholik]

for i := range foos {
    foo := &foos[i]
    ...
}

In a way, this defeats the purpose of "for range". If you can stand an additional line, why not use simple for?

for i := 0; i < len(foos); i++ {

    foo := &foos[i]

    ...

}

 

--
 
 

[David Beaumont]

On 25 September 2012 15:05, Alexei Sholik <alcos...@gmail.com> wrote:
>> for i := range foos {
>> foo := &foos[i]
>> ...
>> }
>
>
> In a way, this defeats the purpose of "for range". If you can stand an
> additional line, why not use simple for?
>
> for i := 0; i < len(foos); i++ {
> foo := &foos[i]
> ...
> }

Exactly how I feel about it. And that's also what's weird about having
the index first (but maybe that's why it's done, because it's
idiomatic to add the extra line).

In all, while the "copy by value" is consistent in the case of the for
loop, I just think it's going to trip up enough people in interesting
ways that we should offer (and clearly document) and alternative that
uses pointers. In fact looping over the pointer should probably be the
suggested default IMO and it's safer and won't bite you if your struct
grows later.

[Steve McCoy]

On Tuesday, September 25, 2012 8:40:25 AM UTC-4, David Beaumont wrote:

I'd prefer:

     for _, &p := range stuff {
         p.fieldName = 7
     }

which I think is unambiguous syntactically.

David

--
David Beaumont :: Îñţérñåţîöñåļîžåţîờñ Libraries :: Google
Google Switzerland GmbH., Brandschenkestrasse 110, CH-8002, Zürich - Switzerland
Tel +41 44 668 1800 :: Fax +41 44 668 1818

Nice, but a little bit weird with the way := normally operates. Maybe "for i, p := &range stuff" ?

[David Beaumont]

On 25 September 2012 16:28, Steve McCoy <mcc...@gmail.com> wrote:
> On Tuesday, September 25, 2012 8:40:25 AM UTC-4, David Beaumont wrote:
>>
>>
>> I'd prefer:
>>
>> for _, &p := range stuff {
>> p.fieldName = 7
>> }
>>
>> which I think is unambiguous syntactically.
>>
>> David
>

> Nice, but a little bit weird with the way := normally operates. Maybe "for
> i, p := &range stuff" ?

Maybe that too, though it's a bit ambiguous when you have maps:

It'd be nice to say:

for k, &v := range myMap { ... }

[Steve McCoy]

On Tuesday, September 25, 2012 10:38:25 AM UTC-4, David Beaumont wrote:

Maybe that too, though it's a bit ambiguous when you have maps:

It'd be nice to say:

  for k, &v := range myMap { ... }

David

--
David Beaumont :: Îñţérñåţîöñåļîžåţîờñ Libraries :: Google
Google Switzerland GmbH., Brandschenkestrasse 110, CH-8002, Zürich - Switzerland
Tel +41 44 668 1800 :: Fax +41 44 668 1818

Ah, good point, although I don't think it could be allowed for maps, since the m[k] expression isn't addressable. But then, if it's no good for maps, I can live without it, period.

[Phil Pennock]

On 2012-09-25 at 15:17 +0200, David Beaumont wrote:
> In all, while the "copy by value" is consistent in the case of the for
> loop, I just think it's going to trip up enough people in interesting
> ways that we should offer (and clearly document) and alternative that
> uses pointers. In fact looping over the pointer should probably be the
> suggested default IMO and it's safer and won't bite you if your struct
> grows later.

And when the range is not returning the original object, but instead a
derived value?

To pick a core language example: strings. str[i] returns the byte at
index i; {{{ for i, rune := range str }}} returns runes created from the
string, rather than original data. As long as range is pass-by-value on
the results, the implementation can avoid memory management issues and
be nice and quick, working with a value in the stack for the conversion
and then return a simple copy of that. The moment you have to consider
the possibility of returning a reference, the implementation needs to
permit the usual Go agnosticism of stack vs heap and the core runtime.

As long as you assume that range can be constrained to only ever be
permitted to return items as they are actually stored, or simple shallow
copies thereof, your suggestion makes sense; but the string library's
API suggests this is not so.

-Phil

[Peter S]

That gives the error: "i declared but not used". In general, whether an actual copy happens is decided by the compiler. If you actually use the value for something, it is likely to be copied to a CPU register at the very least.

I think the discussion here about copying is not really about the cost of copying, but whether that variable can be used to modify the value in the array/slice or not.

If you are "performance conscious", you should profile your application and improve the parts that take the most time. If you feel like you must know in detail what data is copied when and where, you can always look at the assembly output of the compiler using a command such as "go tool 6g -S source.go".

Peter

On Wed, Sep 26, 2012 at 9:48 AM, <bogdan....@gmail.com> wrote:

Actually this is interesting. I didn't realize that it would be a copy either.

Let's say I'm performance conscious. Does the following code copy the value at each iteration step:

for i,_ := range someArray { } 

Thank you.

--
 
 

[Patrick Mylund Nielsen]

Use for i := range foo {...}. I believe it is equivalent to for i := 0; i < len(foo); i++ {...}

On Tue, Sep 25, 2012 at 11:14 PM, <bogdan....@gmail.com> wrote:

I was referring to the "_", is there any copy happening there and then that copy being discarded?

--
 
 

[Nigel Tao]

On 26 September 2012 14:14, <bogdan....@gmail.com> wrote:
> I was referring to the "_", is there any copy happening there and then that
> copy being discarded?

No.

[Nigel Tao]

My mistake; I remembered wrongly. walkrange in
$GOROOT/src/cmd/gc/range.c treats "for i := range b" and "for i, _ :=
range b" differently. The former is preferred.

[David Beaumont]

I agree it's not obviously trivial in all cases.

Perhaps the better answer would be:

Give good compiier warnings/errors (errors would be my preferred way)
for cases where a store is made to a local struct in a loop without
subsequently being read again.

David

[Dan Kortschak]

In the case of i, _ := range ... why is that not optimised to i := range ... (though why someone would write the latter is an interesting question)?

> --
>
>

[Peter]

It's used when assigning to the slice.

//ordered integer array

for i := range a {

    a[i] = i

}

Is that what you meant?

[Nigel Tao]

On 26 September 2012 20:03, Dan Kortschak <dan.ko...@adelaide.edu.au> wrote:
> In the case of i, _ := range ... why is that not optimised to i := range ... (though why someone would write the latter is an interesting question)?

It's simply a compiler bug, although in practice it's trivial to just
write "for i := range".

[Dan Kortschak]

That was pretty much my though; not much drive to fix something that sanity should preclude the use of.

[Daniel Morsing]

On Wed, Sep 26, 2012 at 1:05 PM, Dan Kortschak
<dan.ko...@adelaide.edu.au> wrote:
> That was pretty much my though; not much drive to fix something that sanity should preclude the use of.
>

Regardless, I've submitted a small fix for exactly this issue.

https://codereview.appspot.com/6564052

[Dave Cheney]

This sounds like something that go vet could check for.

> --
>
>

[Ian Lance Taylor]

On Wed, Sep 26, 2012 at 5:37 AM, Dave Cheney <da...@cheney.net> wrote:
> This sounds like something that go vet could check for.

I believe gofmt -s will fix it.

Ian

> On Wed, Sep 26, 2012 at 9:56 PM, Daniel Morsing
> <daniel....@gmail.com> wrote:
>> On Wed, Sep 26, 2012 at 1:05 PM, Dan Kortschak
>> <dan.ko...@adelaide.edu.au> wrote:
>>> That was pretty much my though; not much drive to fix something that sanity should preclude the use of.
>>>
>>
>> Regardless, I've submitted a small fix for exactly this issue.
>>
>> https://codereview.appspot.com/6564052
>>
>> --
>>
>>
>

> --
>
>

[DisposaBoy]

I write latter form in all cases because I find it.clearer.

[minux]

On Sep 27, 2012 1:09 AM, "DisposaBoy" <dispo...@dby.me> wrote:
> I write latter form in all cases because I find it.clearer.

which form?

[DisposaBoy]

for i,_ :=

[mspr...@us.ibm.com]

This just bit me.  I wonder if the discussion above missed the critical point.  Looking at https://golang.org/ref/spec#Address_operators, I see that a variable is considered _addressable_ and that's all there is to it.  Now, in C and other such languages, there is a critical distinction about addresses that is relevant: is the address on the stack or on the heap?  Since the Go doc does not mention this distinction, I wonder if all variables are on the heap.  That would explain why all variables are addressable.  In the example that started this thread, there is only one loop iteration variable; supposing it is on the heap, `&` extracts a pointer to it, and the loop leaves the last value in that variable.

The mental model I had when I wrote my code was something like the following.  I supposed that Go keeps local (to a block in a function) variables on the stack, and the compiler knows this.  A statement like `x = &localVar` is equivalent to something like
```
x = new(<type of localVar>)
*x = localVar
```

(which still seems (to me) to be the right model if we are talking about <constructor> instead of <localVar> --- which is what led me to think this way.)

Mike

[Ian Lance Taylor]

When writing Go you can imagine that all variables are on the heap.

The compiler implements escape analysis optimizations that allocate
variables on the stack when it can be proven to be safe.

Ian

[Wojciech S. Czarnecki]

On Fri, 1 Sep 2017 07:32:34 -0700 (PDT)
mspr...@us.ibm.com wrote:

> that a variable is considered _addressable_ and that's all there is to it.
> Now, in C and other such languages, there is a critical distinction about
> addresses that is relevant: is the address on the stack or on the heap?

"Anytime a value is shared outside the scope of a function’s stack frame, it
will be placed (or allocated) on the heap. It’s the job of the escape
analysis algorithms to find these situations and maintain a level of
integrity in the program." [from:
https://www.goinggo.net/2017/05/language-mechanics-on-escape-analysis.html]

[Or look at an old presentation of Dave Cheney:
https://dave.cheney.net/2014/06/07/five-things-that-make-go-fast]

Hope this helps :)

--
Wojciech S. Czarnecki
<< ^oo^ >> OHIR-RIPE