cgo Handle doc

[Frédéric De Jaeger]

Hi,

There is something that puzzles me a lot in the doc about cgo.Handle there https://pkg.go.dev/runtime/cgo.  It says:

Some C functions accept a void* argument that points to an arbitrary data value supplied by the caller. It is not safe to coerce a cgo.Handle (an integer) to a Go unsafe.Pointer, but instead we can pass the address of the cgo.Handle to the void* parameter, as in this variant of the previous example:

I was under the impression that casting a uintptr to an  unsafe.Pointer just to call a C function (that accepts a  void*) is somewhat valid. (This is not clearly  specified as a valid case in the doc of unsafe.Pointer, so, it might be invalid from a very pedantic point of view).

The advice given in the doc looks worst.  Taking the address of a local variable and giving it to C looks even more undefined (if C stores the pointer and reuse it after the original call returns).

Don't we have a doc issue there ?

I know that in C, the conversion uintptr -> void* is subject to debate (trap representation, ...).  In practice, nobody cares because It just works. (example of the nobody cares https://www.freedesktop.org/software/gstreamer-sdk/data/docs/latest/glib/glib-Type-Conversion-Macros.html)

In practice, I'm almost certain that the naïve code that just gives the cgo.Handle to a C function as a void* is *much* safer that the pattern suggested in the doc.

So should the doc be changed, or am I missing some important piece ?

Thanks

Fred

[Ian Lance Taylor]

It is not safe to convert a uintptr to an unsafe.Pointer when calling
a C function. The docs in the unsafe package are intended to be
precise, and in particular the compiler is aware of them. The
compiler applies special handling to a conversion from unsafe.Pointer
to uintptr in the argument to a function with no body like
syscall.Syscall. That special handling does not occur in calls to cgo
functions. While it will often work in practice, there are cases
where it will fail.

One way to handle this kind of case is to write a tiny C wrapper in
the cgo comment that takes a uintptr and does the conversion to void*
in C. That conversion is safe, because the Go runtime will not be
aware of it and will never mistakenly treat the uintptr value as a
pointer value.

Ian

[Frédéric De Jaeger]

Thanks for your reply.  I was really missing an essential piece.

I found a bit weird that you need to write a helper to pass a uintptr to a void* C function.  It could be nice to allow that

func Foo(x uintptr) {

    C.foo((*C.char)x)

}

but unfortunately, the compiler does not seem to accept it.  Is there a reason ?

Anyway, I suppose the issue is that the runtime will interpret the uintptr, when cast to a unsafe.Pointer as a potential valid pointer and if it looks a bit too much like a legit go pointer, that might confuse the GC.  I suppose that unsafe.Pointer can safely handle C pointers (from the doc of cgo)

If the uintptr contains a valid C pointer, do we have a practical  issue, like with the following code ?

func main() {

   v := uintptr(unsafe.Pointer(C.CString("Hello from stdio")))

   C.myprint(unsafe.Pointer(v))

   C.free(unsafe.Pointer(cs))

}

This is a question about the current implementation, I know it violates the rule of unsafe.Pointer.  This is just to have a better understanding about what the runtime does and when (at conversion time?  C call time ? GC time ?)

From your last paragraph, can I do that ?

/*

#include <stdint.h>

static void* IntToPtr(uintptr_t v) { return (void*)v; }

*/

import "C"

import (

   "runtime/cgo"

   "unsafe"

)

func HandleAsPointer(h cgo.Handle) unsafe.Pointer {

   return C.IntToPtr(C.uintptr_t(uintptr(h)))

}

Here the cast int -> void* happens in the C world and the conversion back to unsafe.Pointer looks legit according to the rule of cgo.  But this will create an unsafe.Pointer containing a totally invalid pointer.  Will it work (practically and theoretically) ?  Is the GC robust against the presence of a random value in an unsafe.Pointer ?

Thanks

[Frédéric De Jaeger]

Given the special cases there https://pkg.go.dev/cmd/cgo#hdr-Special_cases, I would say this is not a good idea.

[Ian Lance Taylor]

On Fri, Nov 25, 2022 at 12:06 AM Frédéric De Jaeger
<fdej...@novaquark.com> wrote:
>
> Thanks for your reply. I was really missing an essential piece.
> I found a bit weird that you need to write a helper to pass a uintptr to a void* C function. It could be nice to allow that
>
> func Foo(x uintptr) {
> C.foo((*C.char)x)
> }
>
> but unfortunately, the compiler does not seem to accept it. Is there a reason ?

In Go you can't convert a uintptr to an arbitrary pointer. You could
write C.foo((*C.char)(unsafe.Pointer(x))). The compiler would accept
that, but it would break the unsafe.Pointer conversion rules.

> Anyway, I suppose the issue is that the runtime will interpret the uintptr, when cast to a unsafe.Pointer as a potential valid pointer and if it looks a bit too much like a legit go pointer, that might confuse the GC. I suppose that unsafe.Pointer can safely handle C pointers (from the doc of cgo)

Yes.

> If the uintptr contains a valid C pointer, do we have a practical issue, like with the following code ?
>
> func main() {
> v := uintptr(unsafe.Pointer(C.CString("Hello from stdio")))
> C.myprint(unsafe.Pointer(v))
> C.free(unsafe.Pointer(cs))
> }
>
> This is a question about the current implementation, I know it violates the rule of unsafe.Pointer. This is just to have a better understanding about what the runtime does and when (at conversion time? C call time ? GC time ?)

Yes, if the pointer is a valid C pointer, then this kind of code is OK
in practice, though forbidden by the rules.

> From your last paragraph, can I do that ?
>
> /*
> #include <stdint.h>
> static void* IntToPtr(uintptr_t v) { return (void*)v; }
> */
> import "C"
>
> import (
> "runtime/cgo"
> "unsafe"
> )
>
> func HandleAsPointer(h cgo.Handle) unsafe.Pointer {
> return C.IntToPtr(C.uintptr_t(uintptr(h)))
> }
>
> Here the cast int -> void* happens in the C world and the conversion back to unsafe.Pointer looks legit according to the rule of cgo. But this will create an unsafe.Pointer containing a totally invalid pointer. Will it work (practically and theoretically) ? Is the GC robust against the presence of a random value in an unsafe.Pointer ?

This won't work. As you say, the GC can't handle an invalid pointer
value that is marked as a pointer.

Ian

[Frédéric De Jaeger]

On Friday, November 25, 2022 at 6:12:01 PM UTC+1 Ian Lance Taylor wrote:

On Fri, Nov 25, 2022 at 12:06 AM Frédéric De Jaeger
>

> Thanks for your reply. I was really missing an essential piece.
> I found a bit weird that you need to write a helper to pass a uintptr to a void* C function. It could be nice to allow that
>
> func Foo(x uintptr) {
> C.foo((*C.char)x)
> }
>
> but unfortunately, the compiler does not seem to accept it. Is there a reason ?

In Go you can't convert a uintptr to an arbitrary pointer. You could
write C.foo((*C.char)(unsafe.Pointer(x))). The compiler would accept
that, but it would break the unsafe.Pointer conversion rules.

I was not asking for a conversion of a uintptr to an arbitrary pointer, that would way too unsafe; but a conversion of  uintptr to an arbitrary C pointer (the same kind of typing rule that currently exists to allow casts  back and forth between uintptr and unsafe.Pointer).  It would have the obvious semantic, with no interference with the runtime.

Seeing other issues like https://groups.google.com/g/golang-nuts/c/wcG1vKnDVkc, that would help in many places.  There is no compatibility issue. I don't think it makes cgo any more dangerous to use.  For that to be fully useful, the compiler would have to accept the type *C.void in a cast expression.

Alternatively, you could provide a type `cgo.Pointer`, that has the same typing rule as unsafe.Pointer, but has no pointer semantic.  But that would be too confusing, I don't like it.

 

> From your last paragraph, can I do that ?
>
> /*
> #include <stdint.h>
> static void* IntToPtr(uintptr_t v) { return (void*)v; }
> */
> import "C"
>
> import (
> "runtime/cgo"
> "unsafe"
> )
>
> func HandleAsPointer(h cgo.Handle) unsafe.Pointer {
> return C.IntToPtr(C.uintptr_t(uintptr(h)))
> }
>
> Here the cast int -> void* happens in the C world and the conversion back to unsafe.Pointer looks legit according to the rule of cgo. But this will create an unsafe.Pointer containing a totally invalid pointer. Will it work (practically and theoretically) ? Is the GC robust against the presence of a random value in an unsafe.Pointer ?

This won't work. As you say, the GC can't handle an invalid pointer
value that is marked as a pointer.

ho.  Then I think that might be good to document in cgo, close to this sentence: "The C type void* is represented by Go's unsafe.Pointer."

that it is illegal to put an invalid pointer in an unsafe.Pointer.

thanks for your answers, this is very enlightening

Fred

[Ian Lance Taylor]

On Fri, Nov 25, 2022 at 12:45 PM Frédéric De Jaeger
<fdej...@novaquark.com> wrote:
>
> I was not asking for a conversion of a uintptr to an arbitrary pointer, that would way too unsafe; but a conversion of uintptr to an arbitrary C pointer (the same kind of typing rule that currently exists to allow casts back and forth between uintptr and unsafe.Pointer). It would have the obvious semantic, with no interference with the runtime.
>
> Seeing other issues like https://groups.google.com/g/golang-nuts/c/wcG1vKnDVkc, that would help in many places. There is no compatibility issue. I don't think it makes cgo any more dangerous to use. For that to be fully useful, the compiler would have to accept the type *C.void in a cast expression.
>
> Alternatively, you could provide a type `cgo.Pointer`, that has the same typing rule as unsafe.Pointer, but has no pointer semantic. But that would be too confusing, I don't like it.

The Go language and Go compiler don't have a notion of a C pointer.
They just know about pointers and other types. If p has type *C.char,
then Go code is permitted to write *p to get a value of type C.char.
That is, a value of type *C.whatever is just a pointer like any other
pointer type. The only thing that is unusual about it is that
C.whatever is defined by C code. And that definition is handled
entirely by the cgo tool. It's not part of the Go language and the Go
compiler doesn't know anything about it.

> Then I think that might be good to document in cgo, close to this sentence: "The C type void* is represented by Go's unsafe.Pointer."
> that it is illegal to put an invalid pointer in an unsafe.Pointer.

Well, maybe. We can't document everything everywhere. The
restriction is not specific to unsafe.Pointer, you can't put an
invalid pointer in any pointer type. There is a separate section in
the cgo docs about passing pointers.

Ian