unsafe string to []byte

[Steve Roth]

The implementation of io.WriteString appears to allocate a new byte slice and copy the string into it:

w.Write([]byte(s))

Many third party libraries avoid the allocation and copy with techniques like:

var b []byte

sh := (*reflect.StringHeader)(unsafe.Pointer(&s))

bh := (*reflect.SliceHeader)(unsafe.Pointer(&b))

bh.Data = sh.Data

bh.Len = sh.Len

bh.Cap = sh.Len

w.Write(b)

I've seen so many different packages do this that it almost seems like a preferred idiom.  Yet, it doesn't seem to be guaranteed safe by the rules in the "unsafe" package documentation; rule 6 comes close to allowing it but doesn't quite get there.  And the fact that the standard library doesn't use it, in an obviously applicable place, is telling.

So, what's the deal here?  Is it safe or not?  Can I use it in my own code?  Must I shun libraries that use it?  (Must I read the source code of every library I use, to see whether it uses it?)

[Axel Wagner]

On Tue, Jul 27, 2021 at 4:15 PM Steve Roth <st...@rothskeller.net> wrote:

The implementation of io.WriteString appears to allocate a new byte slice and copy the string into it:

w.Write([]byte(s))

Only if the writer does not implement `io.StringWriter`. Avoiding this allocation where possible is exactly why `io.StringWriter` exists.

 

Many third party libraries avoid the allocation and copy with techniques like:

var b []byte

sh := (*reflect.StringHeader)(unsafe.Pointer(&s))

bh := (*reflect.SliceHeader)(unsafe.Pointer(&b))

bh.Data = sh.Data

bh.Len = sh.Len

bh.Cap = sh.Len

w.Write(b)

I've seen so many different packages do this that it almost seems like a preferred idiom. Yet, it doesn't seem to be guaranteed safe by the rules in the "unsafe" package documentation; rule 6 comes close to allowing it but doesn't quite get there.  And the fact that the standard library doesn't use it, in an obviously applicable place, is telling.

So, what's the deal here?  Is it safe or not?

No, that code is broken. It makes assumptions about the implementation of `Write`, which is *documented*, but not enforced by the compiler - namely, that `Write` may not retain a reference to the `[]byte` and may not modify its contents. If such an incorrect `io.Writer` is used with a library like this, it might break the program in strange and unforseen ways.

Can I use it in my own code?

There are occasions where it is safe. For example, strings.Builder does a similar thing in a safe way.

So, as with all unsafe: If you know it's safe, it's fine to use. Otherwise, stay away.

  Must I shun libraries that use it?  (Must I read the source code of every library I use, to see whether it uses it?)

Unfortunately there is no way to guarantee that a dependency contains good code.

This particular issue should be reasonably easy to find by grepping for `unsafe`, which is a good practice if you want to avoid potentially unsafe code anyway.

 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAnpqKHoA74DfM5765vUfrH4V6RpBxg1DTkfn3ScN6MhjQwRTQ%40mail.gmail.com.

[Robert Engels]

Agreed. If a Go library uses unsafe I avoid it. 

On Jul 27, 2021, at 9:54 AM, 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfEf6NbkAvDRs6QFO7LDggh-n-6mT3aBnG3gNh678Z6Z1g%40mail.gmail.com.

[Bryan C. Mills]

For what it's worth, my unsafeslice.OfString makes a best effort to detect mutations of the sort that would occur when a Write implementation violates the io.Writer contract.

It allows for vary levels of safety. Under `-race` it successfully detects pretty much every mutation and reports the exact writer goroutine. If instead built with `-tags unsafe` it should produce essentially no overhead compared to the hand-written `reflect.SliceHeader` transformation. The default behavior is somewhere in the middle: less overhead than the race detector, but still paying O(N) overhead in order to detect and report otherwise-inscrutable bugs.

That won't stop an incorrect Writer from breaking your program, but it will at least help you figure out where the violation occurred.

[Robert Engels]

That’s cool. I don’t think most libraries using unsafe go to that level of detail or scrutiny - which can be the source of a lot of subtle failures. 

On Aug 3, 2021, at 3:35 PM, 'Bryan C. Mills' via golang-nuts <golan...@googlegroups.com> wrote:

For what it's worth, my unsafeslice.OfString makes a best effort to detect mutations of the sort that would occur when a Write implementation violates the io.Writer contract.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8dc95c51-9faf-48db-a237-213c32cc00c0n%40googlegroups.com.