Append to slice... what happens?

[Peter Kleiweg]

This:

https://play.golang.org/p/AE670rTMpE

I don't know what I expected, but it's weird. Don't mess with slices.

[Jan Mercl]

On Mon, Jun 27, 2016 at 12:26 AM Peter Kleiweg <pkle...@xs4all.nl> wrote:

> I don't know what I expected, but it's weird. Don't mess with slices.

Well, working as expected, considering slice backing array is possibly shared wrt the result of append. (Or other slice op, b/c slices _are_ values.)

However, it's easy to avoid the append sharing when/where not desirable: https://play.golang.org/p/oIFHamYWB-

--

-j

[Henry]

If you were to change the code a bit as follows https://play.golang.org/p/VwtWRQBrEe , it will work as you expect.

I think it is probably safer to instantiate a slice without specifying the initial capacity.

[Martin Geisler]

Hi Henry,

On Mon, Jun 27, 2016 at 3:55 AM, Henry <henry.ad...@gmail.com> wrote:
> If you were to change the code a bit as follows https://play.golang.org/p/VwtWRQBrEe , it will work as you expect.
>
> I think it is probably safer to instantiate a slice without specifying the initial capacity.

Having the capacity larger than the number of used elements (the
length of the slice) is indeed the problem. However, not specifying
the capacity can also fail you. Here I changed the numbers of elements
appended slightly and print the capacity after each operation:

https://play.golang.org/p/WBTRvgJJKW

As you can see, append will over-allocate and set the capacity of the
underlying array to 4 when we go from an empty slice to a slice with
length 3. This in turn makes the following appends to a and b share
the 4th slot in the array and appending to a will ultimately update
the value you see in b.

In your example you were "lucky" since you started with a = [1, 2] and
cap(a) = 2. When you create b, append notices that the capacity is
exhausted and *creates a new underlying array for b*. After that step,
a and b are no longer "entangled" like this and updates to one cannot
affect the other.

--
Martin Geisler

[Martin Geisler]

To unpack what Jan wrote a little: when slicing, you can specify a
third integer which becomes the capacity of the new slice. So

b := a[:len(a):len(a)]

will create a new slice b with len(b) = len(a) and cap(b) = len(a).
Without the third argument, cap(b) = cap(a). When cap(a) > len(a) you
have the situation where append(a, 123) is quick: is can just insert
the element into the underlying array without having to reallocate
anything.

So the line

b := append(a[:len(a):len(a)], 3, 4)

does two things: it slices a to obtain a new slice with a lower than
normal capacity. Because the capacity exactly matches the length of
the new slice, trying to append to it is guaranteed to reallocate the
underlying array. So append makes a new, larger array and copies
len(a) elements into it. It then writes 3 and 4 into the new array as
well before returning a slice of the array. That slice becomes b.

From that point onwards, a and a are no longer connected via the
underlying array.

BTW, I was about to say that you could simplify the line one step further with

b := append(a[::len(a)], 3, 4)

but that gives a compilation error:

prog.go:11: middle index required in 3-index slice

I wonder what the rationale is for this? It seems inconsistent to me
since the second (middle) index has a useful default (len(a)) that is
used when there are only two indexes used.

--
Martin Geisler

[Dan Kortschak]

On Mon, 2016-06-27 at 07:49 +0200, Martin Geisler wrote:
> BTW, I was about to say that you could simplify the line one step
> further with
>
> b := append(a[::len(a)], 3, 4)
>
> but that gives a compilation error:
>
> prog.go:11: middle index required in 3-index slice
>
> I wonder what the rationale is for this? It seems inconsistent to me
> since the second (middle) index has a useful default (len(a)) that is
> used when there are only two indexes used.

As I remember it, during the design discussions the possibility of using
the shortened syntax you show above was considered, but rejected as an
opening to bug entry (too much semantic weight on a single repeated
character).

[Ian Lance Taylor]

Yes. And, also, the default for the middle index is not wholly
obvious. Should it be len(a) or (new) cap(a)? In your example those
happen to have the same value, but of course in general they do not.

Ian

[Martin Geisler]

Hi Ian,

Hmm, that's a good point that I hadn't considered :-) I figured it
would be len(a) so that

a[x::z] = a[x:len(a):z]

That keeps the defaults intact from normal 2-index slicing.

For the indices to be in range, you must have z >= y where y = len(a).
So maybe one could argue that the middle index y could be clamped at
z, that is, the middle index y could be set to min(z, len(a)). That
way you can have a slice with length 10 and still write

a[2::4] = a[2:min(4, 10):4] = a[2:4:4]

where there indices are all in range.

I see how this becomes more complicated than I initially thought... is
that the kind of mess you were thinking of? :-)

--
Martin Geisler

[Ian Lance Taylor]

Yes, that is an example. If there is a default, it should be obvious
in all cases what the default should be. If it's not obvious, there
shouldn't be a default.

Ian

[rog...@gmail.com]

Can anyone ELI5 it to me, please ? Why if we remove cap result is different ? I don't understand that.

[Ian Lance Taylor]

On Sun, Jul 3, 2016 at 4:27 AM, <rog...@gmail.com> wrote:
> Can anyone ELI5 it to me, please ? Why if we remove cap result is different
> ? I don't understand that.

https://blog.golang.org/slices

Ian

[rog...@gmail.com]

Thx Ian,

So basically it's because we allocate new array in version without cap, right?  Because it outgrows current cap. So pointer from slice b is pointing to new slice (i've checked they have different addresses for first element) different from a.
And in version with cap=10 we still append to the same array as there is no relocation, that's why we have same elements in both slices (they point to the same array). Right?

[rog...@gmail.com]

It seems i can't edit my original post so:
There should be "So pointer from slice b is pointing to new array"  instead of "So pointer from slice b is pointing to new slice"

[Martin Geisler]

Hi Rogue,

On Sun, Jul 3, 2016 at 5:13 PM, <rog...@gmail.com> wrote:
> It seems i can't edit my original post so:

While it may not look like it from if you use the Google Groups
interface, emails are being sent out in the background when you post
to golang-nuts. So there's no way to edit anything, you can only send
follow up messages.

> There should be "So pointer from slice b is pointing to new array" instead
> of "So pointer from slice b is pointing to new slice"

Yes, that is correct.

> W dniu niedziela, 3 lipca 2016 17:05:06 UTC+2 użytkownik rog...@gmail.com
> napisał:
>>
>> Thx Ian,
>>
>> So basically it's because we allocate new array in version without cap,
>> right? Because it outgrows current cap. So pointer from slice b is pointing
>> to new slice (i've checked they have different addresses for first element)
>> different from a.
>> And in version with cap=10 we still append to the same array as there is
>> no relocation, that's why we have same elements in both slices (they point
>> to the same array). Right?

You got it!

The capacity of the slice tells you how much more room there is beyond
the length. If you set the capacity to the length, the next call to
append will conclude that there's no space in the underlying array and
proceed to allocate a new array, copy the data from the slide to the
new array, and return a slice into this new array. This breaks the
"link" between the old and new slices since they now refer to
different arrays.

--
Martin Geisler

[rog...@gmail.com]

Thank you Martin!