It seems that go 1.22 has optimized the implementation of string to byte slicing and no longer requires memory allocation.

289 views
Skip to first unread message

fliter

unread,
Nov 27, 2023, 7:12:54 AM11/27/23
to golang-nuts
It seems that go 1.22 has optimized the implementation of string to byte slicing and no longer requires memory allocation.


But why not optimize byte slicing to string conversion together?


```go
package main

import (
"reflect"
"testing"
"unsafe"
)


func BenchmarkString(b *testing.B) {
byteSli := []byte{123, 34, 100, 101, 102, 97, 117, 108, 116, 34, 58, 123, 34, 99, 111, 109, 109, 111, 110, 34, 58, 123, 34, 112, 101, 116, 34, 58, 123, 34, 102, 105, 118, 101, 34, 58, 34, 230, 150, 145, 230, 150, 145, 34, 44, 34, 102, 111, 117, 114, 34, 58, 34, 231, 154, 174, 231, 147, 156, 231, 147, 156, 34, 44, 34, 111, 110, 101, 34, 58, 34, 229, 188, 165, 229, 188, 165, 230, 135, 181, 34, 44, 34, 116, 104, 114, 101, 101, 34, 58, 34, 229, 145, 134, 229, 145, 134, 34, 44, 34, 116, 119, 111, 34, 58, 34, 233, 187, 132, 230, 169, 153, 230, 169, 153, 34, 125, 44, 34, 114, 101, 108, 97, 116, 105, 111, 110, 34, 58, 123, 34, 102, 97, 116, 104, 101, 114, 34, 58, 34, 99, 117, 105, 120, 120, 120, 120, 120, 120, 120, 34, 44, 34, 109, 111, 116, 104, 101, 114, 34, 58, 34, 121, 105, 110, 120, 120, 120, 120, 120, 34, 44, 34, 119, 105, 102, 101, 34, 58, 34, 112, 101, 110, 103, 120, 120, 34, 125, 125, 125, 125}

_ = string(byteSli)

}

func BenchmarkUnsafe(b *testing.B) {
byteSli := []byte{123, 34, 100, 101, 102, 97, 117, 108, 116, 34, 58, 123, 34, 99, 111, 109, 109, 111, 110, 34, 58, 123, 34, 112, 101, 116, 34, 58, 123, 34, 102, 105, 118, 101, 34, 58, 34, 230, 150, 145, 230, 150, 145, 34, 44, 34, 102, 111, 117, 114, 34, 58, 34, 231, 154, 174, 231, 147, 156, 231, 147, 156, 34, 44, 34, 111, 110, 101, 34, 58, 34, 229, 188, 165, 229, 188, 165, 230, 135, 181, 34, 44, 34, 116, 104, 114, 101, 101, 34, 58, 34, 229, 145, 134, 229, 145, 134, 34, 44, 34, 116, 119, 111, 34, 58, 34, 233, 187, 132, 230, 169, 153, 230, 169, 153, 34, 125, 44, 34, 114, 101, 108, 97, 116, 105, 111, 110, 34, 58, 123, 34, 102, 97, 116, 104, 101, 114, 34, 58, 34, 99, 117, 105, 120, 120, 120, 120, 120, 120, 120, 34, 44, 34, 109, 111, 116, 104, 101, 114, 34, 58, 34, 121, 105, 110, 120, 120, 120, 120, 120, 34, 44, 34, 119, 105, 102, 101, 34, 58, 34, 112, 101, 110, 103, 120, 120, 34, 125, 125, 125, 125}

_ = *(*string)(unsafe.Pointer(&byteSli))
}

func BenchmarkByteStyle(b *testing.B) {

str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd","two":"黄ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

_ = []byte(str)

}

func BenchmarkWithUnsafe(b *testing.B) {

str := `{"default":{"common":{"pet":{"five":"aa","four":"bb","one":"cc","three":"dd","two":"黄ee"},"relation":{"father":"ff","mother":"mm","wife":"ww"}}}}`

sh := (*reflect.StringHeader)(unsafe.Pointer(&str))
bh := reflect.SliceHeader{
Data: sh.Data,
Len:  sh.Len,
Cap:  sh.Len,
}
_ = *(*[]byte)(unsafe.Pointer(&bh))

}
```

```bench_test.go
package main

import (
"testing"
)


func BenchmarkTest1(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkString(b)
}
}

func BenchmarkTest2(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkUnsafe(b)
}
}

func BenchmarkTest3(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkByteStyle(b)
}
}

func BenchmarkTest4(b *testing.B) {
for i := 0; i < b.N; i++ {
BenchmarkWithUnsafe(b)
}
}
```

when go version is 1.21

```shell
 go version
go version go1.21.0 darwin/arm64

goos: darwin
goarch: arm64
pkg: bc
BenchmarkTest1-8        37078008                32.45 ns/op          192 B/op          1 allocs/op
BenchmarkTest2-8        144106840                9.181 ns/op           0 B/op          0 allocs/op
BenchmarkTest3-8        38973375                28.94 ns/op          192 B/op          1 allocs/op
BenchmarkTest4-8        1000000000               0.3130 ns/op          0 B/op          0 allocs/op
PASS
ok      bc      6.038s

```



when go version is 1.22

```shell
 go version                        
go version devel go1.22-631a6c2abf Fri Nov 17 23:34:11 2023 +0000 darwin/arm64


 go test -test.bench=".*" -benchmem
goos: darwin
goarch: arm64
pkg: bc
BenchmarkTest1-8        35727334                33.51 ns/op          192 B/op          1 allocs/op
BenchmarkTest2-8        147172425                8.157 ns/op           0 B/op          0 allocs/op
BenchmarkTest3-8        1000000000               0.3136 ns/op          0 B/op          0 allocs/op
BenchmarkTest4-8        1000000000               0.3153 ns/op          0 B/op          0 allocs/op
PASS
ok      bc      5.095s
```



Volker Dobler

unread,
Nov 27, 2023, 9:34:51 AM11/27/23
to golang-nuts
On Monday, 27 November 2023 at 13:12:54 UTC+1 fliter wrote:
But why not optimize byte slicing to string conversion together?

Try to implement that conversion and you'll see.
Note that strings are immutable, even if converted
from a byte slice (which are by their very nature)
mutable. Good luck in automatic proving that no
other thing mutates the converted-from slice.

V. 

Volker Dobler

unread,
Nov 27, 2023, 9:36:56 AM11/27/23
to golang-nuts
Ah, and one more:

On Monday, 27 November 2023 at 13:12:54 UTC+1 fliter wrote:
BenchmarkTest1-8        35727334                33.51 ns/op          192 B/op          1 allocs/op
BenchmarkTest2-8        147172425                8.157 ns/op           0 B/op          0 allocs/op
BenchmarkTest3-8        1000000000               0.3136 ns/op          0 B/op          0 allocs/op
BenchmarkTest4-8        1000000000               0.3153 ns/op          0 B/op          0 allocs/op

Try to reflect upon what 0.3 ns/op means _actually_ on the
hardware level.

V 

peterGo

unread,
Nov 27, 2023, 10:14:03 AM11/27/23
to golang-nuts

Assigning Benchmark results to package variables:

var (
    str string
    byt []byte
)

BenchmarkString
str = string(byteSli)

BenchmarkUnsafe
str = *(*string)(unsafe.Pointer(&byteSli))

BenchmarkByteStyle
byt = []byte(str)

BenchmarkWithUnsafe
byt = *(*[]byte)(unsafe.Pointer(&bh))

$ go1.21 test nuts_test.go -run=! -bench=. -benchmem
BenchmarkTest1-12  22507272    50.72 ns/op   192 B/op  1 allocs/op
BenchmarkTest2-12  27105592    49.92 ns/op   192 B/op  1 allocs/op
BenchmarkTest3-12  25006983    41.32 ns/op   160 B/op  1 allocs/op
BenchmarkTest4-12  1000000000   1.158 ns/op    0 B/op  0 allocs/op

$ go1.22 test nuts_test.go -run=! -bench=. -benchmem
BenchmarkTest1-12  17386624    65.79 ns/op   192 B/op  1 allocs/op
BenchmarkTest2-12  18603463    62.30 ns/op   192 B/op  1 allocs/op
BenchmarkTest3-12  23741634    57.76 ns/op   160 B/op  1 allocs/op
BenchmarkTest4-12  1000000000   1.157 ns/op    0 B/op  0 allocs/op

peter

tapi...@gmail.com

unread,
Nov 28, 2023, 2:08:30 AM11/28/23
to golang-nuts
The tip compiler is able to detect some simple string->[]byte cases in which duplication is not needed.

fliter

unread,
Nov 28, 2023, 6:41:19 AM11/28/23
to golang-nuts
Thanks！

fliter

unread,
Nov 28, 2023, 6:43:56 AM11/28/23
to golang-nuts
Thank you so much!
Message has been deleted

fliter

unread,
Nov 28, 2023, 6:45:26 AM11/28/23
to golang-nuts
Why does BenchmarkTest3 in go 1.22 still have memory allocation? My local test doesn’t have it
Reply all
Reply to author
Forward
0 new messages