if passing pointers to cgo is (will be?) forbidden, what is the correct way ?

[nicolas...@gmail.com]

Preamble
========
I am reading Issue 8310 "cmd/cgo: do not let Go pointers end up in C"
  https://code.google.com/p/go/issues/detail?id=8310

  #0 rsc says:
  > We need some way to prevent Go pointers from crossing the boundary into C.

  #6 dvyukov says:
  > Updating Go objects from C code never worked. C code runs concurrently with GC.
  > GC needs to see consistent picture of the world. If C updates Go objects, GC corrupts heap.

  #16 ian says:
  > In the long run there just isn't any way that we can support passing a Go pointer into C.
  > We want to make it possible to write a moving garbage collector, but that means that the garbage collector has to be able to change all pointers.
  > The GC won't be able to see pointers stored in C code.  So we can't permit that.

Question 1)
===========
Let's have these two pieces of code:

  var x [10]byte
  C.f(&x[0])      // f modifies the bytes in x

or

  var x C.T       // in C file: typedef struct T {unsigned char  data[10];} T;
  C.f(&x)         // f modifies the bytes in x

I understand that with a future moving GC, if GC runs when f() is running, f() will corrupt heap.
But is dvyukov saying that the code above is already broken now with Go 1.3, even if C code doesn't change any pointer in the object but just plain bytes ?


In http://golang.org/cmd/cgo/, I read:
 "In Go, you must pass the pointer to the first element explicitly: C.f(&x[0])."
If this idiom may not be supported in the future, shouldn't a warning be already included in this documentation ?

Question 2)
===========

> #15 r... says:
> Passing a Go heap pointer to C can break today's 1.3 GC which does not scan C stacks / variables.

If passing pointers &x[0] is already a problem in Go 1.3, I will change all my code right now.
But I don't understand this stack/variables issue. Can someone explain this point ?


Question 3)
===========

If passing &x[0] is not recommended, the only official means to communicate between Go world and C heap is to use:
  func C.CString(string) *C.char
  func C.GoString(*C.char) string
  func C.GoStringN(*C.char, C.int) string
  func C.GoBytes(unsafe.Pointer, C.int) []byte

  or wrapping C array in struct, so that they can be passed by value between Go and C, like this ?

    http://play.golang.org/p/5ck0C_NKVx

  I noticed that because a pointer is passed, the object always escape to heap. So, a copy is better for performance than creating garbage on heap, in the end.

[nkatsaros]

I've got the same exact questions. My other issue is user pointers for callbacks where I need to reference a go object. Right now I'm using a map[uintptr]*T as mentioned in https://code.google.com/p/go/issues/detail?id=8310#c18 but I don't know how safe it is or how long it will be safe for.

I think a blog post on this topic would be valuable for anyone writing bindings for C libraries in Go.

[Emily Maier]

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Your example in Question 1 may or may not work fine depending on the
rest of your code. What Q2 means is that when the Go GC runs, it doesn't
look on the C heap for references to Go objects. So if you pass a Go
object to C and the Go side no longer has any references, it might get
freed while the C code is still using it. Here's an example of what
would be unsafe currently:

C.f(C.CString("foo"))

As far as getting data from Go to C goes, I have a (very experimental)
library at https://github.com/emilymaier/cmemory that implements
Reader/Writer/etc buffers with C allocations underneath. So data can be
written to it and passed to C safely.

Emily

[Ian Taylor]

On Sun, Nov 9, 2014 at 12:53 PM, <nicolas...@gmail.com> wrote:
>
> Question 1)
> ===========
> Let's have these two pieces of code:
>
> var x [10]byte
> C.f(&x[0]) // f modifies the bytes in x
>
> or
>
> var x C.T // in C file: typedef struct T {unsigned char data[10];}
> T;
> C.f(&x) // f modifies the bytes in x
>
> I understand that with a future moving GC, if GC runs when f() is running,
> f() will corrupt heap.
> But is dvyukov saying that the code above is already broken now with Go 1.3,
> even if C code doesn't change any pointer in the object but just plain bytes
> ?

This kind of code will work in Go 1.4. You are correct that the kind
of problem Dmitriy is discussing only arises if the C code changes a
pointer value. In 1.3 and 1.4 there is no issue with a *byte.

In future Go releases there will be a problem, because we want to
leave open the possibility of a moving GC, in which case this kind of
code will stop working. I think it will be possible to have cgo
detect and reject some obvious cases. It may also be possible in some
cases to have cgo automatically copy the memory into a safe area
before passing it to the C code. I'm not sure think that can work in
general, though.

> In http://golang.org/cmd/cgo/, I read:
> "In Go, you must pass the pointer to the first element explicitly:
> C.f(&x[0])."
> If this idiom may not be supported in the future, shouldn't a warning be
> already included in this documentation ?

I'll send a CL to fix that case.

We haven't written down rules in the docs because we haven't settled
what the rules are. These are issues 8839 and 8310.

> Question 2)
> ===========
>
>> #15 r... says:
>> Passing a Go heap pointer to C can break today's 1.3 GC which does not
>> scan C stacks / variables.
>
> If passing pointers &x[0] is already a problem in Go 1.3, I will change all
> my code right now.
> But I don't understand this stack/variables issue. Can someone explain this
> point ?

It will break if you don't keep a pointer in Go code. If you are
careful to ensure that Go code holds a copy of the pointer, you will
be OK with 1.3 and 1.4.

> Question 3)
> ===========
>
> If passing &x[0] is not recommended, the only official means to communicate
> between Go world and C heap is to use:
> func C.CString(string) *C.char
> func C.GoString(*C.char) string
> func C.GoStringN(*C.char, C.int) string
> func C.GoBytes(unsafe.Pointer, C.int) []byte
>
> or wrapping C array in struct, so that they can be passed by value between
> Go and C, like this ?
>
> http://play.golang.org/p/5ck0C_NKVx
>
> I noticed that because a pointer is passed, the object always escape to
> heap. So, a copy is better for performance than creating garbage on heap, in
> the end.

Simply using a struct does not help. In your code the pointer to the
struct is still a Go pointer. Using a struct would only help if you
called into C to get memory to write into. That is an addition to
your list above: call a C function that calls malloc and returns a C
pointer. You still can't store a Go pointer into that C pointer, but
you can safely store any non-pointer values.

Ian

[Ian Taylor]

On Sun, Nov 9, 2014 at 4:20 PM, nkatsaros <ni...@nickkatsaros.com> wrote:
>
> I've got the same exact questions. My other issue is user pointers for
> callbacks where I need to reference a go object. Right now I'm using a
> map[uintptr]*T as mentioned in
> https://code.google.com/p/go/issues/detail?id=8310#c18 but I don't know how
> safe it is or how long it will be safe for.

It will always be safe to pass uintptr values in and out of C. Using
a map to convert the uintptr values to Go pointers should always work
reliably. What you will have to be careful about is how you get the
uintptr that you pass to C. One issue that can arise, in principle,
is that you might have two different Go pointers produce the same
uintptr value, because the first Go pointer moved and a second one was
allocated at the same address as where the first started.

> I think a blog post on this topic would be valuable for anyone writing
> bindings for C libraries in Go.

Yes, we will certainly do that when we know what the rules will be.
We wanted to get them sorted out for 1.4, but we failed.

Ian

[nicolas riesch]

Ian, thank you very much for all this clarification :-)))

Please bear with me, I have two last questions.

Question A)
===========

Just to be sure, in the code below, I pass struct by value, and the returned value is also a value.
As there is no pointer involved, I imagine it will always be safe ?

package main

/*

typedef struct T {
    unsigned char  data[10];
} T;

T set(T t, int val) {
    int i;

    for (i=0; i<10; i++) {
        t.data[i] = (unsigned char)val;
    }

    return t;
}

*/
import "C"

func main() {
  var a C.T

  a = C.set(a, C.int(2))   // the argument a is passed by value to C.set(), and the returned value is copied back into a. I imagine it is safe ?
}

Question B)
===========

This little program creates a Go slice attached to a mallocated byte array in C heap, and writes the content directly from Go.
Is it allowed, or frown upon ? I dont' see any other way of writing directly from Go into C memory...

   http://play.golang.org/p/i1M713OSni

Thank you for your patience.

[Ian Taylor]

On Mon, Nov 10, 2014 at 4:06 PM, nicolas riesch
<nicolas...@gmail.com> wrote:
>
> Just to be sure, in the code below, I pass struct by value, and the returned
> value is also a value.
> As there is no pointer involved, I imagine it will always be safe ?
>
> package main
>
> /*
> typedef struct T {
> unsigned char data[10];
> } T;
>
> T set(T t, int val) {
> int i;
>
> for (i=0; i<10; i++) {
> t.data[i] = (unsigned char)val;
> }
>
> return t;
> }
>
> */
> import "C"
>
> func main() {
> var a C.T
>
> a = C.set(a, C.int(2)) // the argument a is passed by value to C.set(),
> and the returned value is copied back into a. I imagine it is safe ?
> }

This is fine.

> This little program creates a Go slice attached to a mallocated byte array
> in C heap, and writes the content directly from Go.
> Is it allowed, or frown upon ? I dont' see any other way of writing directly
> from Go into C memory...
>
> http://play.golang.org/p/i1M713OSni

This should be OK.

I don't think it's a good idea to use reflect.SliceHeader. It was a
mistake for that type to ever be exported. We only keep it because of
the Go 1 guarantee.

A better way to turn a C pointer into a Go slice is something like

func Cbyteslice(p unsafe.Pointer, n int) []byte {
return (*[0x7fffffff]byte)(p)[:n]
}

Ian

[robert....@gmail.com]

I am currently writing C bindings for one of my Go projects and am uncertain how to deal with issues 8839 and 8310. I see that the release-go1.5 label was removed from issue 8839 at the end of last year. Is it safe to assume that cgo will work as-is in Go 1.5, e.g. passing a Go-backed byte slice to C works like in 1.4?

Beyond version 1.5, are there any indications on future use of cgo that are more recent than the ones already outlined before in the issues or this thread?

Many thanks in advance,

Robert

From the discussions I understand that passing a Go-backed byte slice to C might not be supported in the future. 

[Ian Lance Taylor]

On Mon, Mar 9, 2015 at 1:50 AM, <robert....@gmail.com> wrote:
>
> I am currently writing C bindings for one of my Go projects and am uncertain
> how to deal with issues 8839 and 8310. I see that the release-go1.5 label
> was removed from issue 8839 at the end of last year. Is it safe to assume
> that cgo will work as-is in Go 1.5, e.g. passing a Go-backed byte slice to C
> works like in 1.4?
> Beyond version 1.5, are there any indications on future use of cgo that are
> more recent than the ones already outlined before in the issues or this
> thread?

These are good questions but unfortunately we don't have good answers
at this time.

Ian