0 length slices and arrays in golang

[Anmol Sethi]

Was trying to figure out how 0 length arrays and slices behaved in Golang. Came up with two snippets of code (I found the code somewhere and modified it a bit to work with this)

https://play.golang.org/p/ew2YYgvpGC

https://play.golang.org/p/jm2p6L6WCG

I learnt from the website that nil arrays ([]int(nil)) have a pointer value of nil, so I decided to test it. Sure enough, that is the case. I'm just confused on make and slicing an array. It has unexpected behaviour for me.

I am really confused by the behaviour of these two. The first one runs fine on my computer and on the playground. I've noticed that the address of the first and last array is always the exact same? Why?

Why is this?

The second one is weirder. This is the exact same code as the previous one, except there are snippets of other code for the len/cap in between. It does not run on the go playground, there is an error at the last one with the sliced array, for some reason, the slice gets a length of 3 (on my computer it is 0 for the last one, and the cap for all of them is 272851504). It does run on my computer however. I noticed that the address of the first array created with make is always smaller than the last one. Its always different, and a bit smaller (the first one), why? There is no change in the code for the address of the array

Also, why does make() even create an array? How does an array of length 0 even look in memory?

[Axel Wagner]

I find your testcode hard to read and I am unsure, if it does, what it
is supposed to do. A simpler to read alternative:
https://play.golang.org/p/6331ay7tC4
doesn't exhibit the same behavior you are describing. Which leads me to
suspect, that the problem is with your code, not with the actual
observations :)

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Volker Dobler]

None of your code is an array, you just use slices.

I doubt that "inspecting" slices with unsafe is a useful way

to _learn_ something about slices.

I'd reccommend http://blog.golang.org/slices, never think

about a slice as an array and do not use unsafe, it is unsafe.

If you want to print the capacity or length of a slice just use

the cap and len builtins and all is fine.

V.

[Anmol Sethi]

It does? 

    // Create a pointer to the underlying array

    addPtr := (*[0]int)(unsafe.Pointer(*(*uintptr)(address)))

that creates a pointer to the array of the slice? And outputs its address. But this address's behaviour is whats confusing me. Two functions return completely different behaviour even though they do the exact same thing, one just as a bit of code between to output other stuff....

That make([]int, 0) creates a slice with a pointer to something. What is this something because the array is 0, it takes up 0 bytes, so what is this pointing to?

[James Bardin]

On Monday, March 23, 2015 at 4:24:53 PM UTC-4, Anmol Sethi wrote:

It does? 

    // Create a pointer to the underlying array

    addPtr := (*[0]int)(unsafe.Pointer(*(*uintptr)(address)))

that creates a pointer to the array of the slice? And outputs its address.

After combining the twi lines, you're code is

    

    addPtr := (*[0]int)(unsafe.Pointer(*(*uintptr)(unsafe.Pointer(&slice))))

All this does take the address of the slice header, then covert it to an unsafe pointer, then to a *uintptr, then an unsafe pointer, and finally an *[0]int. A slice header is not an *[0]int.

If you want the address to the array, you could just use

    addrPtr := &slice[0]

 

But this address's behaviour is whats confusing me. Two functions return completely different behaviour even though they do the exact same thing, 

No they're not. See the answer you got on your original StackOverflow question.

 

That make([]int, 0) creates a slice with a pointer to something.

No it doesn't

What is this something because the array is 0, it takes up 0 bytes, so what is this pointing to?

Nothing, the pointer is nil.

 

[Axel Wagner]

Hi Anmol,

after experimenting a bit, I am now convinced, that the effects you are
seeing are due to inlining and optimization. The reason is, that, if in:
https://play.golang.org/p/fdkFYTzx5I
I see the expected behavior, while if you comment out lines 29 and 30:
https://play.golang.org/p/aBkKMvk3Rk
You get the behavior you are experiencing. I am unsure, whether it is a
bug or not, that you can observe this behavioral difference (on the one
hand this code uses unsafe, which is, well, unsafe. On the other hand,
you might get similar results when doing a fmt.Printf("%p", &arr[0])).

Again, the way I do the printing of the slice header is the "canonical"
way to do that and it will give you a better way to understand *this
particular implementation* of slices.

Best,

Axel

[James Bardin]

On Mon, Mar 23, 2015 at 5:00 PM, Anmol Sethi <anmol....@gmail.com> wrote:

Try that piece of code, the addPtr is definitely pointing to the array, not the slice header.

Oh yes, I missed one dereference in that chain of calls. It is because you're dereferencing the address of the slice header as a uintptr, essentially reading the value if the first field which also happens to be a uintptr. 

[Anmol Sethi]

It is definitely pointing to the array, not the slice

package main




import (
    "fmt"
    "unsafe"
    "reflect"
)


func main() {
    var slice []int
//    slice = make([]int, 0)
//    InspectSlice(slice)
//    slice = []int(nil)
//    InspectSlice(slice)
    array := [3]int{1,3,4}
    slice = array[0:3]
    InspectSlice(slice)
}


func InspectSlice(slice []int) {
    // Capture the address to the slice structure
    address := unsafe.Pointer(&slice)


    // Capture the address where the length and cap size is stored
    lenAddr := uintptr(address) + uintptr(8)
    capAddr := uintptr(address) + uintptr(16)


    // Create pointers to the length and cap size
    lenPtr := (*int)(unsafe.Pointer(lenAddr))
    capPtr := (*int)(unsafe.Pointer(capAddr))

    // Create a pointer to the underlying array

    addPtr := (*[3]int)(unsafe.Pointer(*(*uintptr)(address)))


    fmt.Printf("Slice Addr[%p] Len Addr[0x%x] Cap Addr[0x%x]\n",
    address,
    lenAddr,
    capAddr)


    fmt.Printf("Slice Length[%d] Cap[%d]\n",
    *lenPtr,
    *capPtr)


    for index := 0; index < *lenPtr; index++ {
        fmt.Printf("[%d] %p %d\n",
        index,
        &(*addPtr)[index],
        (*addPtr)[index])
    }
    fmt.Printf("%p\n", addPtr)
    fmt.Println(reflect.TypeOf(*addPtr))
    fmt.Printf("\n\n")
}

[Anmol Sethi]

Yes, so are my points valid now? Why is the function changing it? It does nothing different... Why does the first function give me both the same, then the second gives the first one smaller, third one bigger. I'm assuming there is some internal reason on how Go is handled at the playground, thus my code doesn't work properly there. And what array does the make(int[], 0) point to? Why does it have a value..? I guess the performance thing could be it, but i don know.

[Axel Wagner]

The behavior you are observing is *definitely* due to compiler optimizations. Observe the difference when running your code with vs. without compiler optimizations:
http://sprunge.us/jMIO

[Anmol Sethi]

You're right. So the compiler is optimizing when the code is larger? Why does this make the address so small in the first one from you, verse now? You know what, maybe i'm getting too low level, perhaps only the engineers of the compiler and golang could probably answer my questions. But even then. Does anyone know the answer to why make(int[], 0) comes up with an pointer with some sort of value?

package main


import (
    "fmt"
    "unsafe"

)


func main() {
    var slice []int

    slice = make([]int, 0)

    InspectSlice(slice)
    slice = []int(nil)
   // InspectSlice(slice) ERROR HERE BECAUSE IT IS A NIL POINTER

}


func InspectSlice(slice []int) {
    // Capture the address to the slice structure
    address := unsafe.Pointer(&slice)

    // Create a pointer to the underlying array

    addPtr := (*[0]int)(unsafe.Pointer(*(*uintptr)(address)))

    fmt.Printf("%v\n", *addPtr)
}

I ran this code, and it shows me that there is in fact a 0 length array. It outputs it fine. Now how does this array look in memory with this 0 value? You can see that with nil, there is an error. Therefore, what I am saying makes sense! make([]int, 0) creates an empty array. Not a nil pointer like the nil slice. 

[Axel Wagner]

I assume the difference is less size and more (maybe) the cast to uintptr in your code (but not in my original simplification) and interfaces. In my code, the pointer stays a pointer, in your case it gets cast to a uintptr, which makes it a valid assumption that this pointer isn't used. Or something like that, I am just guessing here :)

To answer your question about x := make([]int, 0) creating a valid value: x needs to be distinct from nil. len and cap obviously have to be 0. If the pointer would also be 0, x would be indistinguishable of []int(nil). Therefore, the compiler assigns a bogus-value to the pointer-field. How exactly it does this (i.e. by reserving an address in the data segment or by hashing the typename, or by pointing to the type-definition, or by just ponting to some constant bogus address…) is implementation dependent. You can observe the same behavior with
http://play.golang.org/p/0NTnHsdQqB
Both return values of new need to be distinct from nil. The relevant part of the spec is it's last sentence:
http://golang.org/ref/spec#Size_and_alignment_guarantees
"A struct or array type has size zero if it contains no fields (or elements, respectively) that have a size greater than zero. Two distinct zero-size variables may have the same address in memory."

[Axel Wagner]

An amended example:
http://play.golang.org/p/qSrIXq6C-K
(I am surprised, that %p works with slices in this way, but apparently it does *shrug*)

[atd...@gmail.com]

You've chosen the wrong size for you pointers (on the playground).

https://play.golang.org/p/BszdYBc9d6

Also remember that Go is pass by value. So if you pass a pointer to a function, you will actually use a copy that point to the same thing. This copy is stored at a different location. The allocations are often close enough to be done in contiguous blocks. That's why you see the address as being only slightly different.

If you need to refer to the same ref type, pass a slice pointer.

As far as the memory representation for the 0 length array, I guess a fixed address is being chosen to represent 0 length arrays. (just a guess).

[Bakul Shah]

On Mon, 23 Mar 2015 14:40:41 PDT Anmol Sethi <anmol....@gmail.com> wrote:
>
> You're right. So the compiler is optimizing when the code is larger? Why
> does this make the address so small in the first one from you, verse now?
> You know what, maybe i'm getting too low level, perhaps only the engineers
> of the compiler and golang could probably answer my questions. But even
> then. Does anyone know the answer to why make(int[], 0) comes up with an
> pointer with some sort of value?

Perhaps this 2009 article "Go Data Structures" by Russ Cox
will help (no idea if it describes the current Go
implementation).

http://research.swtch.com/godata

[Anmol Sethi]

by both return values, you mean the two structs pointers coming from New? They need to be different from a nil one? Ah I see what you mean with the spec and all. That makes a lot more sense now. I see now, so it needs to be distinct so that it can share the memory with another zero size variable. Thanks for all the help man, really appreciated it!

[Anmol Sethi]

You're correct about the pointers, thanks, you cleared that up. But read the previous responses, this doesn't actually have anything to do with pass by value. Take a closer look at the statement in which I take a pointer to the underlying array from the slice header. 

[Anmol Sethi]

yea its weird, it prints the address of the array.

[Anmol Sethi]

already read it, it helped quite a bit

[Anmol Sethi]

what do you mean by "I assume the difference is less size and more (maybe) the cast to uintptr in your code (but not in my original simplification) and interfaces. In my code, the pointer stays a pointer, in your case it gets cast to a uintptr, which makes it a valid assumption that this pointer isn't used. Or something like that, I am just guessing here :)"

What do you mean the pointer isn't used? What about casting it to uintptr does this? And my method returns the exact same as yours, the simplified one, with the other function, the longer one. With that in mind, i think I can confidently say it most likely is not a result of casting to uintptr. har, oh well, its way too low level. I have no idea where to start to even figure it out. But whatever, thanks for all the help.

Sorry for the spam btw, whoops lol

[Axel Wagner]

Hi,

Anmol Sethi <anmol....@gmail.com> writes:
> What do you mean the pointer isn't used? What about casting it to uintptr
> does this? And my method returns the exact same as yours, the simplified
> one, with the other function, the longer one. With that in mind, i think I
> can confidently say it most likely is not a result of casting to
> uintptr.

I was randomly guessing. I read on golang-dev or on golang-nuts or
somewhere, that (at least) the garbage collector doesn't consider a
uintptr an address, so if e.g. you save an address in a uintptr and then
later convert it back to an address, the GC might have moved or
collected the memory allready and the address might be invalid. I was
randomly guessing, that the compiler uses a similar heuristic, to
determine, that the memory of your slices isn't actually used and so it
will optimize the allocation out. It was a random guess.

I now am relatively certain, that it has to do with interface-calls and
how the compiler does escape-analysis. See for example
https://play.golang.org/p/PZadRQ1nkc
which is again a simplified version of your program (and it does the
same). If however, you uncomment line 17, you now get the expected
result. Note that we did nothing different, except actually using the
pointed-to memory.

My suspicion (again, random guess) is, that in case of the
reflect.SliceHeader-cast the compiler now couldn't proof, that the
memory wasn't actually used (because the Printf-call is an interface
call, so it might use the pointer? or something) and thus needed to do
the allocation (resp. pointing to the constant
zero-size-pointer). Compare e.g. the behavior of line 24 and line 25 in
https://play.golang.org/p/D6b-fiWJvW
Both do essentially the same thing, but in case of line 24 the compiler
knows completely, that you are only accessing the Data-field of the
slice-header as a uintptr, where in line 25 Printf could *potentially*
do more than that.

But all of this is again, random guessing.

> har, oh well, its way too low level.

Yes. Yes it is :) No go programmer should need to care about this stuff.

> I have no idea where to start to even figure it out.

From my random poking at what the compiler does for different things, I
would start having a look at how it does escape analysis, because that
is definitely related. At the end of the day though, I think that a
couple of different optimization passes are interacting here and it is
probably very hard to fully explain the behavior, without having a
relatively deep insight into the compiler.

> But whatever, thanks for all the help.

You are welcome.