uintptr to Go object, unsafe to pass to any function, including syscalls/C functions?

[Daniel Eloff]

I'm trying to understand the new rules introduced in 1.3 with the typed GC, it looks like uintptr is unsafe to pass to any function now. From a related question in May:

Ian Lance Tayler wrote:

Yes, stack copying can be an issue in unusual cases.  If you write 
    var a [5]int 
    p = unsafe.Pointer(&a[0]) 
    v := uintptr(p) 
    f() 
    p = unsafe.Pointer(v) 
then it is possible that the stack will have been copied during the 
call to f, which could in some cases mean that p will no longer be 
pointing at a. 
As long as you retain a pointer in Go, it should be fine to convert a 
Go pointer to uintptr and pass it to C.  It doesn't matter what C does 
with the value.  That said, if you have a pointer value, it would in 
general be better to pass an unsafe.Pointer to C rather than a 
uintptr. 

From my limited (and quite possibly wrong) understanding, that last paragraph seems incorrect. It's unsafe to do anything that could grow the stack between storing a pointer in a uintptr and converting it back to an unsafe.Pointer.

let's say you have function f. It could be a C function or a syscall or a Go function, it doesn't matter.

func f(p uintptr)

the you call it:

var a [5]int

p = unsafe.Pointer(&a[0]) 
v := uintptr(p) 

f(v)

This is the same as Ian's example? If the call to f() grows the stack, v could point to nonsense and f() could segfault or worse. It's less obvious, but I think:

f(uintptr(unsafe.Pointer(&a[0])))

Has the same problem. Is this really a problem, or am I missing something here?

Thanks,

Dan

[Ian Lance Taylor]

It's not a problem if f is a Go function. If the GC can move a stack,
it will update all Go pointers into the stack, one way or another.
That is possible because we know everything about pointers in Go code.

It is a problem if f is a C function or a system call. That is a
problem we need to figure out one way or another.

http://golang.org/issue/8310
http://golang.org/issue/7192

Ian

[Daniel Eloff]

> var a [5]int
> p = unsafe.Pointer(&a[0])
> v := uintptr(p)
> f(v)
>

 

It's not a problem if f is a Go function.  If the GC can move a stack,

it will update all Go pointers into the stack, one way or another.
That is possible because we know everything about pointers in Go code.

But here the pointer is temporarily in a uintptr for the call, which Go treats as an integer. So it won't update it and the process can crash. It seems like just about anytime you convert a pointer into a uintptr, it's unsafe.

Currently now if it's a pointer to the stack, and the stack changes size, it doesn't matter if your uintptr is only a temporary, it will cease to have meaning. e.g. uintptr(unsafe.Pointer(&a[0])) + getOffset() could be unsafe if getOffset() is evaluated second and causes the stack to grow.

In the future a moving GC would break all conversions to uintptr, because a GC can always happen between converting to a uintptr and using it. So even uintptr(unsafe.Pointer(...)) + x could become unsafe in the future, although that would probably be unacceptable for Go (good luck writing Go in Go, and making it perform well, without pointer arithmetic.) Basically the GC treating uintptr as an integer makes it a minefield of potential problems. One solution might be to consider uintptr as a pointer for purposes of GC, or eliminate the need for it by allowing pointer arithmetic on unsafe.Pointer directly. Some solution that will be forward compatible with a moving GC will be needed.

Pointer arithmetic is a necessary evil for any kind of low-level or systems code, and Go is supposed to be good at those kinds of problems. I have 3K lines of Go and climbing fast, that use pointer arithmetic heavily. I'm really betting the future of my company on the language. It's the kind of thing one would have previously used C for, but there are so many advantages to using Go and just suffering the messy syntax when pointer manipulations are required.

Cheers,

Dan

 

[Carlos Castillo]

Ian, are you saying that in a call to a f(uintptr) function, the stack slot for the uintptr argument will be updated (as if it were actually a pointer) if the object is moved by the GC?

EG: http://play.golang.org/p/I67OCTnF-4

If the GC is run during do_stuff, the address stored in the uintptr argument will be updated correctly to match the new location of the heap alloc'd int?

[DV]

On Wednesday, July 23, 2014 3:21:34 PM UTC-6, Daniel Eloff wrote:

> var a [5]int
> p = unsafe.Pointer(&a[0])
> v := uintptr(p)
> f(v)
>

 

It's not a problem if f is a Go function.  If the GC can move a stack,
it will update all Go pointers into the stack, one way or another.
That is possible because we know everything about pointers in Go code.

But here the pointer is temporarily in a uintptr for the call, which Go treats as an integer. So it won't update it and the process can crash. It seems like just about anytime you convert a pointer into a uintptr, it's unsafe.

Currently now if it's a pointer to the stack, and the stack changes size, it doesn't matter if your uintptr is only a temporary, it will cease to have meaning. e.g. uintptr(unsafe.Pointer(&a[0])) + getOffset() could be unsafe if getOffset() is evaluated second and causes the stack to grow.

In the future a moving GC would break all conversions to uintptr, because a GC can always happen between converting to a uintptr and using it. So even uintptr(unsafe.Pointer(...)) + x could become unsafe in the future, although that would probably be unacceptable for Go (good luck writing Go in Go, and making it perform well, without pointer arithmetic.) Basically the GC treating uintptr as an integer makes it a minefield of potential problems. One solution might be to consider uintptr as a pointer for purposes of GC, or eliminate the need for it by allowing pointer arithmetic on unsafe.Pointer directly. Some solution that will be forward compatible with a moving GC will be needed.

Pointer arithmetic is a necessary evil for any kind of low-level or systems code, and Go is supposed to be good at those kinds of problems. I have 3K lines of Go and climbing fast, that use pointer arithmetic heavily

I'm not familiar with your company or your particular product, so I apologize in advance for saying so, but isn't "too much pointer arithmetic" in Go usually some form of red flag? Go purposefully does *not* allow pointer arithmetic without importing "unsafe" and doing the conversions manually; in my opinion - pointer arithmetic in Go is difficult and cumbersome because it's generally discouraged. Pointer arithmetic being difficult is by design. 

Are you just writing "C" in Go? People new to Go tend to try to mash other languages' style and idioms into Go, and only end up getting frustrated. 

There's a "Go" way of doing things where "unsafe" and pointer arithmetic are usually (99% of the time?) unnecessary or not desired.  

There's currently an interesting discussion between rsc and dvyukov where rsc states bluntly that readability and code maintenance in Go is more important to him than raw performance:

https://groups.google.com/d/topic/golang-dev/SBmIen68ys0/discussion

Basically:

1. Try to write Go code the "Go way" (which might not be the most "performant" way)
   
2. If you absolutely, 120%, no-two-ways-about-it must have certain features like easy low-level pointer arithmetic, generics, etc, then, sadly, maybe Go isn't for you?

Just some stuff to ponder on

[Daniel Eloff]

On Wednesday, July 23, 2014 4:47:12 PM UTC-5, DV wrote:

I'm not familiar with your company or your particular product, so I apologize in advance for saying so, but isn't "too much pointer arithmetic" in Go usually some form of red flag? Go purposefully does *not* allow pointer arithmetic without importing "unsafe" and doing the conversions manually; in my opinion - pointer arithmetic in Go is difficult and cumbersome because it's generally discouraged. Pointer arithmetic being difficult is by design. 

We're developing a highly-concurrent in-memory database.

 

Are you just writing "C" in Go? People new to Go tend to try to mash other languages' style and idioms into Go, and only end up getting frustrated. 

Yes, that's exactly what we're doing, for the code where it matters. In fact, the current query engine written in Go is designed to be replaced by a SIMD query engine with the core routines in assembly. Go can do that, either by calling C (too much overhead in Go for my taste, even if it can be amortized by making fewer, chunkier calls) or directly with .S files, which is probably the way we'll go, excuse the pun.

 

There's currently an interesting discussion between rsc and dvyukov where rsc states bluntly that readability and code maintenance in Go is more important to him than raw performance:

https://groups.google.com/d/topic/golang-dev/SBmIen68ys0/discussion

I've been following that discussion, because I've specifically had the same issue that Dmitry did. I'm with Dmitry on the performance vs readability trade off. I will happily make the 10% of the code, where performance really matters, nearly as unreadable as it needs to be in order to squeeze every last drop of performance out of the hardware. For the rest of the code, the 90% of it really, I prefer readability and maintainability like rsc. Inflexible hard and fast stances on anything are an ideology, I'm an engineer, I like to keep it practical. Dmitry and rsc might fall on different sides of the spectrum, but I think that discussion is proof that they both are practical, engineer type people too. rsc, in that discussion tried to find a compromise solution that would make both him and Dmitry happy, that's what good engineering is all about.

I'm aware that sharing memory directly instead of communicating, manually managing "native" memory, and using lots of pointer arithmetic and low-level tricks is allowed by Go, but frowned upon. That's fine, I frown on those things too, but I'm a practical man, and I'll use them when I have no other option.

Whether Go was the best fit for the project is tough to say. The available libraries are a perfect fit. The tools are fantastic (missing only a debugger.) The language itself makes writing, readable, maintainable code easier while still performing close enough to C. And it makes the hard things possible, not ideal, but possible, and that's what counts.

Other options considered were:

C/C++: horrible for development speed, safety, readability, maintainability and compile times

Rust: too immature and lacking in needed libraries, also too complex making it harder to write readable, maintainable code

Scala: too complex, too many ways to make unreadable, unmaintainable code, slow compiler (was the runner up choice to Go)

Haskell: next

Erlang: fundamentally opposed to one big shared address space

Python/Ruby: too slow, too high level, prefer statically typed for big projects

LuaJIT: fast, maybe faster than Go for what we're doing, but too small a community, missing needed libraries, very dependent on Mike Pall (large hit-by-a-bus risk)

Java: frankly it just crushes my soul to have to program in Java, I would not do it unless I had no other option

C#: platform lock-in

D: missing needed libraries, the community is very small and that's not likely to change

As things stand, Go seems like the best choice, the gamble being that it may one day prevent some of the low-level things we do. But in the worst case we can always hack the runtime, it is open source after-all, and a compiler will always have a way internally to do low-level things. If we're using .S files anyway with custom assembly routines, just about anything is possible.

Cheers,

Dan

[k...@golang.org]

I think Ian was incorrect when he stated that it would be ok.

var a [5]int

p = unsafe.Pointer(&a[0]) 

v := uintptr(p) 

f(v)

q := unsafe.Pointer(v)

..use q..

is bad, whether f is in Go or C.  If you're going to keep a pointer across across a function call, keep it in a pointer, not an integer.  i.e. use unsafe.Pointer, not uintptr.

Casting a uintptr to an unsafe.Pointer is dangerous.  The only safe thing you can do with it is unsafe.Pointer(uintptr(p) + offset).  We'll guarantee that this idiom continues to work, at least until/if we have a specific API for unsafe pointer arithmetic.

[Ian Lance Taylor]

On Wed, Jul 23, 2014 at 2:21 PM, Daniel Eloff <dan....@gmail.com> wrote:
>> > var a [5]int
>> > p = unsafe.Pointer(&a[0])
>> > v := uintptr(p)
>> > f(v)
>> >
>
>
>>
>> It's not a problem if f is a Go function. If the GC can move a stack,
>> it will update all Go pointers into the stack, one way or another.
>> That is possible because we know everything about pointers in Go code.
>
>
> But here the pointer is temporarily in a uintptr for the call, which Go
> treats as an integer. So it won't update it and the process can crash. It
> seems like just about anytime you convert a pointer into a uintptr, it's
> unsafe.

I'm sorry, you're quite right. I didn't read your code properly.
You're right: converting a pointer to uintptr and passing that uintptr
to a function is not safe. You need to keep it as unsafe.Pointer.

> In the future a moving GC would break all conversions to uintptr, because a
> GC can always happen between converting to a uintptr and using it. So even
> uintptr(unsafe.Pointer(...)) + x could become unsafe in the future, although
> that would probably be unacceptable for Go (good luck writing Go in Go, and
> making it perform well, without pointer arithmetic.) Basically the GC
> treating uintptr as an integer makes it a minefield of potential problems.
> One solution might be to consider uintptr as a pointer for purposes of GC,
> or eliminate the need for it by allowing pointer arithmetic on
> unsafe.Pointer directly. Some solution that will be forward compatible with
> a moving GC will be needed.

This is where we need rules about when it's OK to convert to uintptr,
which is the point of http://golang.org/issue/7192 . I think that we
clearly have to permit unsafe.Pointer(uintptr(p) + c), though it's
conceivable that we will have to implement it as unsafe.Offset(p, c).

Ian

[Daniel Eloff]

I'm sorry, you're quite right.  I didn't read your code properly. 
You're right: converting a pointer to uintptr and passing that uintptr 
to a function is not safe.  You need to keep it as unsafe.Pointer.  

Ok, so unsafe is really unsafe now if it holds a GC pointer and you want to do pointer arithmetic with it. You have to be very careful with it, much more so than unsafe code in C.

This is where we need rules about when it's OK to convert to uintptr,
which is the point of http://golang.org/issue/7192 .  I think that we
clearly have to permit unsafe.Pointer(uintptr(p) + c), though it's
conceivable that we will have to implement it as unsafe.Offset(p, c).

Honestly, unsafe.Offset(p, index*size) is more readable than  unsafe.Pointer(uintptr(p) + uintptr(index) * size), and it would continue working with a moving GC, whereas it's difficult to see how the other could. If it were implemented like the special runtime append, it could take a typed pointer and return a pointer of the same type. That would make working with it a lot cleaner.

Conversion to/from (mostly from) uintptr would still work for "native" memory, although one could argue that unsafe.Pointer could be used instead.

Thanks for taking the time to clear things up for me.

Dan

[Russ Cox]

Run go vet on your code. If it's happy, your code is fine. If not, your code is probably NOT fine.

Russ

[capnm]

vet isn't happy with those code fragments (a "possible misuse of unsafe.Pointer")

func PtrOffset(offset int) unsafe.Pointer {

->return unsafe.Pointer(uintptr(offset))

}

func Ptr(data interface{}) unsafe.Pointer {

if data == nil {

return unsafe.Pointer(nil)

}

var addr uintptr

v := reflect.ValueOf(data)

switch v.Type().Kind() {

case reflect.Ptr:

e := v.Elem()

switch e.Kind() {

case

reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,

reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,

reflect.Float32, reflect.Float64:

addr = e.UnsafeAddr()

}

case reflect.Uintptr:

addr = v.Pointer()

case reflect.Slice:

addr = v.Index(0).UnsafeAddr()

case reflect.Array:

addr = v.UnsafeAddr()

default:

panic(fmt.Sprintf("Unsupproted type %s; must be a pointer, slice, or array", v.Type()))

}

->return unsafe.Pointer(addr)

}

It is not entirely clear to me: is such code save in Go1.3?

If not, could someone think up a scenario where it breaks? 

Thanks,

Martin

[Ian Lance Taylor]

On Tue, Aug 5, 2014 at 1:38 PM, capnm <cap...@gmail.com> wrote:
>
> vet isn't happy with those code fragments (a "possible misuse of
> unsafe.Pointer")
>
> func PtrOffset(offset int) unsafe.Pointer {
> ->return unsafe.Pointer(uintptr(offset))
> }

This code is not safe in 1.3 if offset holds a pointer value.

> func Ptr(data interface{}) unsafe.Pointer {
> if data == nil {
> return unsafe.Pointer(nil)
> }
> var addr uintptr
> v := reflect.ValueOf(data)
> switch v.Type().Kind() {
> case reflect.Ptr:
> e := v.Elem()
> switch e.Kind() {
> case
> reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
> reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,
> reflect.Float32, reflect.Float64:
> addr = e.UnsafeAddr()
> }
> case reflect.Uintptr:
> addr = v.Pointer()
> case reflect.Slice:
> addr = v.Index(0).UnsafeAddr()
> case reflect.Array:
> addr = v.UnsafeAddr()
> default:
> panic(fmt.Sprintf("Unsupproted type %s; must be a pointer, slice, or array",
> v.Type()))
> }
> ->return unsafe.Pointer(addr)
> }

This code looks safe in 1.3, but may break in future releases. You
should change the type of addr to be unsafe.Pointer.

Ian

[Eric Woroshow]

> This code is not safe in 1.3 if offset holds a pointer value. 

And if offset instead holds an arbitrary integer offset (i.e., as a type of "index" into a block of memory)? Can the GC handle unsafe.Pointer values containing invalid pointers?

> You  should change the type of addr to be unsafe.Pointer. 

The addr value is returned by the reflect package, which is explicit about returning uintptr to force callers to convert to unsafe.Pointer themselves. Is there an alternative mechanism for extracting an unsafe.Pointer value from reflect?

[Ian Lance Taylor]

On Tue, Aug 5, 2014 at 3:31 PM, Eric Woroshow <er...@ericw.ca> wrote:
>> This code is not safe in 1.3 if offset holds a pointer value.
> And if offset instead holds an arbitrary integer offset (i.e., as a type of
> "index" into a block of memory)? Can the GC handle unsafe.Pointer values
> containing invalid pointers?

Yes.

>> You should change the type of addr to be unsafe.Pointer.
> The addr value is returned by the reflect package, which is explicit about
> returning uintptr to force callers to convert to unsafe.Pointer themselves.
> Is there an alternative mechanism for extracting an unsafe.Pointer value
> from reflect?

You should write
addr = unsafe.Pointer(e.UnsafeAddr())

Ian

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.