efficiency of string <-> []byte conversion

[Alex Flint]

As of go1.8, do conversions between strings and byte slices always generate a copy?

I understand the immutability guarantees in the language spec, but I'm interested in this from an efficiency standpoint. I can imagine a compiler that analyzes whether a byte slice created from such a conversion is ever modified and foregoing the copy in some cases, while still adhering to the immutability guarantees.

I have searched this forum for a past thread on this topic but have come up empty. Feel free to point me to the relevant discussion.

[Val]

Hello Alex,
good point. I remember this has indeed come up in discussions in this forum.

Searching "byte slice string copy" gave (among others) [1] [2] [3] [4] [5] [6] [7]. All of these look related to the suggested optimization "don't allocate&copy for a []byte to string, or string to []byte, conversion, under certain circumstances".
Cheers
 Val

[1] https://groups.google.com/forum/#!searchin/golang-nuts/byte$20slice$20string$20copy|sort:relevance/golang-nuts/m46jaS6bgZg/R5hSOPFz5m0J
[2] https://groups.google.com/forum/#!searchin/golang-nuts/byte$20slice$20string$20copy|sort:relevance/golang-nuts/CwoKDOhv6vw/sf35VZOZDQAJ
[3] https://groups.google.com/forum/#!searchin/golang-nuts/byte$20slice$20string$20copy|sort:relevance/golang-nuts/ENgbUzYvCuU/gWa8V8wGAwAJ
[4] https://groups.google.com/forum/#!searchin/golang-nuts/byte$20slice$20string$20copy|sort:relevance/golang-nuts/ajXzEM6lqJI/vRsqXxizBAAJ
[5] https://groups.google.com/forum/#!searchin/golang-nuts/byte$20slice$20string$20copy|sort:relevance/golang-nuts/f2cX-BIJsqY/bzK77xpU8PMJ
[6] https://groups.google.com/forum/#!searchin/golang-nuts/byte$20slice$20string$20copy|sort:relevance/golang-nuts/IfILC-wllaI/U8v88T3aBgAJ
[7] https://groups.google.com/forum/#!searchin/golang-nuts/byte$20slice$20string$20copy|sort:relevance/golang-nuts/fXmG83gZOJg/eixtKSQiNysJ

[Alex Flint]

Thanks Val. I actually did a search before posting this and turned up many of those same links, but it seems like they're all just reasoning from the immutability of strings, which I don't think is quite sufficient to answer the question in all cases. Anyway, thanks for your help and I think for now I'll work on the assumption that string/byteslice conversions do always copy.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/C6BBNTRSEwg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Val]

Right, most approaches don't cover all the cases (which would be impossible, but we can still try to tend to).
In my porous memories, some cases were not exactly about immutability, rather about escape analysis :
"If I just built []byte x, and I return string(x), and x doesn't escape, then the conversion shoud be free of charge"
and symmetrically though less useful :
"If I just built string y, and I return []byte(y), and y doesn't escape, then the conversion shoud be free of charge"

But I don't have skills to answer these kind of compiler-related question, so I would be more comfortable letting others speak.
Cheers

[Ian Lance Taylor]

On Mon, Feb 13, 2017 at 12:31 PM, Alex Flint <alex....@gmail.com> wrote:
>
> As of go1.8, do conversions between strings and byte slices always generate
> a copy?

Usually but not absolutely always.

The gc compiler has an optimization for map lookups. For a
map[string]T, when s is a []byte, m[string(s)] will not make a copy.

I'm not aware of any other similar slice <-> string optimization in
the gc compiler. There could be some that I don't know about.

Ian

[mart...@uos.de]

some more uses:

string([]byte):

// Some internal compiler optimizations use this function.

// - Used for m[string(k)] lookup where m is a string-keyed map and k is a []byte.

// - Used for "<"+string(b)+">" concatenation where b is []byte.

// - Used for string(b)=="foo" comparison where b is []byte.

https://github.com/golang/go/blob/master/src/runtime/string.go#L125
https://github.com/golang/go/blob/master/src/cmd/compile/internal/gc/walk.go#L1448

the other way around:

[]byte(string)
// The only such case today is:
// for i, c := range []byte(string)

https://github.com/golang/go/blob/master/src/cmd/compile/internal/gc/walk.go#L1485

Martin