Possible Go Compiler or Runtime bug?

[Kyle Harrity]

I first asked this on https://reddit.com/r/golang but the contribution guide on github recommends this forum, so I'll post here before finally raising an issue on github if this appears to be a real bug.

ORIGINAL POST:

I came across this issue in some code I was reviewing today where a string is converted into a []byte and then a 32 byte slice is taken from that and returned. It returns a 32 byte slice even if the string is empty or less than 32 bytes in length as long as its not a string literal (comes from a function or stored in variable). I can index the slice normally and iterate over its elements, but attempting to print it with fmt.Printf causes a runtime error where it realizes the capacity is not actually 32. Trying to get a slice larger than 32 fails though smaller slices are okay. I think that has something to do with the storage needed to describe a slice 8 bytes for memory location, 8 bytes for size, 8 bytes for capacity, 8 for padding as explained here: https://stackoverflow.com/questions/67839752/why-does-an-empty-slice-have-24-bytes

Here's a playground demo: https://play.golang.com/p/yiLPvRYq8PJ

Maybe this is a known issue and or expected behavior so I thought I'd ask here before raising an issue on github.

[Brian Candler]

That looks very weird. The panic is triggered if I uncomment line 17 (the final fmt.Printf) even though it never reaches there - it panics when getStrBytes is called from line 9 (in line 23).

[Brian Candler]

Interesting:

https://play.golang.com/p/nSZ7tKSeot4

With the Printf commented out, it shows t has a cap of 32.  With Printf uncommented, the cap drops to 0.  Maybe the slice buffer is allocated on the stack in one case, and the heap on another?

The slice header of 24 bytes shouldn't have anything to do with this. The cap() of a slice relates to the amount of storage allocated for the data elements (which the header points to), not the space consumed by the header itself.

[Michael Knyszek]

The number 32 for the capacity comes from this constant, and the returned byte slice in that case is likely coming from a compiler-inserted call to stringtoslicebyte. An empty string is correctly replaced by a zero-length byte slice, it just so happens that the capacity of the array backing that slice may or may not be zero. As Brian said, commenting or uncommenting that line likely just leads to the byte slice's backing array being allocated differently.

I believe this is working as intended, because I don't think the spec makes any guarantees about the capacity of the slice you get back from a conversion. On the one hand, it is a little surprising given that the make builtin very explicitly sets the capacity of the slice. On the other hand, append is an example of a built-in that may return a slice with a larger capacity than requested, so it's not like this case is the only outlier.

[Michael Knyszek]

On Thursday, July 27, 2023 at 6:16:57 PM UTC-4 Michael Knyszek wrote:

I believe this is working as intended, because I don't think the spec makes any guarantees about the capacity of the slice you get back from a conversion.

 (Other than the fact that the capacity must always be >= the length.)

[Ian Lance Taylor]

On Thu, Jul 27, 2023 at 3:17 PM 'Michael Knyszek' via golang-nuts
<golan...@googlegroups.com> wrote:
>
> The number 32 for the capacity comes from this constant, and the returned byte slice in that case is likely coming from a compiler-inserted call to stringtoslicebyte. An empty string is correctly replaced by a zero-length byte slice, it just so happens that the capacity of the array backing that slice may or may not be zero. As Brian said, commenting or uncommenting that line likely just leads to the byte slice's backing array being allocated differently.
>
> I believe this is working as intended, because I don't think the spec makes any guarantees about the capacity of the slice you get back from a conversion. On the one hand, it is a little surprising given that the make builtin very explicitly sets the capacity of the slice. On the other hand, append is an example of a built-in that may return a slice with a larger capacity than requested, so it's not like this case is the only outlier.

I agree that this is not a bug. The language makes no promises about
the capacity of the slice returned by a string to []byte conversion.

The comment in the code

//this crashes, panics because capacity is 0, even though I can access
all 32 bytes in the loop above

is misleading. In the case that crashes, the program can't access all
32 bytes in the loop above.

Ian

[wagner riffel]

On 27/07/2023 03:48, Kyle Harrity wrote:
> Maybe this is a known issue and or expected behavior so I thought I'd
> ask here before raising an issue on github.
>

I think I know what's going on, the compiler inlined getStrBytes and can
prove "k" is short enough to put in the stack, then the runtime creates
the slice backed from an array on the stack, but never sets its
capacity, just length, so you're left with 32 bytes of capacity to
expand (see
https://go.googlesource.com/go/+/refs/tags/go1.20.6/src/runtime/string.go#170).
When you uncomment your print of k, then the compiler can't prove that
fmt.Printf won't keep the address of k, so it's not safe to store it on
the stack anymore, and you have the "expected behavior", an array
allocated from rawbyteslice which does set the capacity to len(s) (0 in
this case).

In the linked code above, changing "b = buf[:len(s):len(s)]" should
"fix" this, it's quoted because I did grasp through the spec and
couldn't find anything about which is the capacity of a slice created
from a conversion of a string, as far as I can tell both programs are valid.

[tapi...@gmail.com]

On Friday, July 28, 2023 at 6:44:21 AM UTC+8 wagner riffel wrote:

On 27/07/2023 03:48, Kyle Harrity wrote:
> Maybe this is a known issue and or expected behavior so I thought I'd
> ask here before raising an issue on github.
>
I think I know what's going on, the compiler inlined getStrBytes and can
prove "k" is short enough to put in the stack, then the runtime creates
the slice backed from an array on the stack, but never sets its
capacity, just length, so you're left with 32 bytes of capacity to
expand (see
https://go.googlesource.com/go/+/refs/tags/go1.20.6/src/runtime/string.go#170).
When you uncomment your print of k, then the compiler can't prove that
fmt.Printf won't keep the address of k, so it's not safe to store it on
the stack anymore, and you have the "expected behavior", an array
allocated from rawbyteslice which does set the capacity to len(s) (0 in
this case).

Note that even if it is not stored on the stack, the behavior is still not always "expected".

When it is stored on heap. the capacity will be aligned to a nearby memory class size

(even this is also implementation specific).