CGO - Possible to pass double pointers from Golang to C and receive data back?

[loc.n...@sentient.ai]

Hi,

I am exploring CGO right now, and I was wondering if it is possible to allocate data from C and return it back to Go via a function double pointer argument?

//int GetData(ContextType** context)
//{
//     *context = ...
//     return errorcode;
//}
import "C"

func main() {
     error = C.GetData(  <what goes here?>  )
}

Is this allowed?  I know that returning ContextType* in the return value is fine, but sometimes you want to get allocated data via function arguments.

Thanks

[Ian Lance Taylor]

With the new garbage collector you can not pass a Go pointer into C
code, so the kind of code you are suggesting is not permitted.

Have your function return a C struct.

Ian

[jto...@gmail.com]

It's probably to late to speak up about this, but the unintended side effects of this change to Go are tragic.

Specifically, it is now impossible to implement an io.Reader or io.Writer using Cgo without doing a memory copy. 

For a specific use case, this means that github.com/spacemonkeygo/openssl is going to require memory copies for all transport data when it totally doesn't need to.

Is there any chance some kind of memory pinning for a []byte buffer is on the roadmap (I admit I can't imagine how it would work)? Passing in a []byte buffer into C as a data source/target is tremendously useful performance-wise.

-JT

[Luna Duclos]

What will the new proper way be of passing Go byte pointers to C ? Will a CBytes function or similar be added to cgo?

--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dmitry Vyukov]

There were no final decision on this. And we do pass Go pointers to
syscalls in std lib (syscall.Read/Write are notable examples). Copying
all data passed to syscall.Read/Write does not look like a feasible
option.

[Dmitry Vyukov]

On Wed, Feb 25, 2015 at 1:34 PM, Dmitry Vyukov <dvy...@google.com> wrote:
>>>
>>> I am exploring CGO right now, and I was wondering if it is possible to
>>> allocate data from C and return it back to Go via a function double pointer
>>> argument?
>>>
>>> //int GetData(ContextType** context)
>>> //{
>>> // *context = ...
>>> // return errorcode;
>>> //}
>>> import "C"
>>>
>>> func main() {
>>> error = C.GetData( <what goes here?> )
>>> }
>>>
>>> Is this allowed? I know that returning ContextType* in the return value is
>>> fine, but sometimes you want to get allocated data via function arguments.
>>
>> With the new garbage collector you can not pass a Go pointer into C
>> code, so the kind of code you are suggesting is not permitted.
>
>
> There were no final decision on this. And we do pass Go pointers to
> syscalls in std lib (syscall.Read/Write are notable examples). Copying
> all data passed to syscall.Read/Write does not look like a feasible
> option.

But double pointers are most likely not OK, because they break
"copying semantics".

[David Crawshaw]

On Wed, Feb 25, 2015 at 5:35 AM, 'Dmitry Vyukov' via golang-dev
<golan...@googlegroups.com> wrote:
> But double pointers are most likely not OK, because they break
> "copying semantics".

The last advice offered was: don't worry about cgo.

https://groups.google.com/forum/#!msg/golang-dev/qvOqcmAkKnA/CcdW-ogVgrYJ

Given the wide range of existing cgo usage (in particular, passing
structures containing long-lived pointers to C), I don't think
speculating on future cgo restrictions right now is useful.

[Maxim Khitrov]

We should just develop software now that may possibly, but probably
won't, but maybe, though almost certainly not break in some indefinite
future?

We're being told not to worry, but the new GC is coming and
definitely, under no circumstances, should we be passing Go pointers
into C... Slightly conflicting information. It would be nice if the Go
team made a final decision about the direction to take with cgo. If
you want to stick to that restriction, ok, but then you may as well
get rid of cgo altogether. A C interface that doesn't allow you to
share Go-allocated data or pass Go pointers back to Go via C is going
to be useless to all but toy applications. The interface is
inefficient enough as it is.

JT said it best, "side effects of this change to Go are tragic."

[Ian Lance Taylor]

On Wed, Feb 25, 2015 at 12:29 AM, <jto...@gmail.com> wrote:
>
> It's probably to late to speak up about this, but the unintended side
> effects of this change to Go are tragic.
>
> Specifically, it is now impossible to implement an io.Reader or io.Writer
> using Cgo without doing a memory copy.
>
> For a specific use case, this means that github.com/spacemonkeygo/openssl is
> going to require memory copies for all transport data when it totally
> doesn't need to.
>
> Is there any chance some kind of memory pinning for a []byte buffer is on
> the roadmap (I admit I can't imagine how it would work)? Passing in a []byte
> buffer into C as a data source/target is tremendously useful
> performance-wise.

We still have not pinned down the exact rules for this, and I
sincerely apologize for that. We've been distracted by other things.

However, let's not confuse the inability to pass Go pointers into C
code with the handling of buffers. I don't see any way that we can
support code such as the OP described, in which we pass a random Go
pointer into C code and the C code changes the memory to which that
pointer points. But passing buffers of data back and forth, data
which definitely does not contain pointers, is different. That may be
feasible. We don't know yet.

An approach that should certainly work is to allocate the byte buffers
in C and write into them in Go. This is certainly inconvenient, as
certain Go idioms such as using append to grow a slice will not work.
But it should be safe.

We're very aware of the issues with cgo, and we'll do our best to find
approaches that will work.

Ian

[Ian Lance Taylor]

On Wed, Feb 25, 2015 at 12:29 AM, <jto...@gmail.com> wrote:
>

> It's probably to late to speak up about this, but the unintended side
> effects of this change to Go are tragic.
>
> Specifically, it is now impossible to implement an io.Reader or io.Writer
> using Cgo without doing a memory copy.
>
> For a specific use case, this means that github.com/spacemonkeygo/openssl is
> going to require memory copies for all transport data when it totally
> doesn't need to.
>
> Is there any chance some kind of memory pinning for a []byte buffer is on
> the roadmap (I admit I can't imagine how it would work)? Passing in a []byte
> buffer into C as a data source/target is tremendously useful
> performance-wise.

[jto...@gmail.com]

We still have not pinned down the exact rules for this, and I
sincerely apologize for that.  We've been distracted by other things.

That's totally fine. It's encouraging to hear it's not as cut and dry as "expect-every-memory-range-to-be-moved-unexpectedly".

 

An approach that should certainly work is to allocate the byte buffers
in C and write into them in Go.  This is certainly inconvenient, as
certain Go idioms such as using append to grow a slice will not work.
But it should be safe.

Yeah, I've been considering this, but it does put a pretty heavy burden on, again, something like implementing io.Reader. Who knows who allocated the target byte slice. Using C.malloc to create slices also has the unfortunate side effect of complicating bookkeeping under reslicing (so that you can eventually free the memory).

 

We're very aware of the issues with cgo, and we'll do our best to find
approaches that will work.

Awesome, thanks. 

As a personal anecdote for why I've been in particular worried about this, we for some reason never got around to upgrading from 1.2 to 1.3, and when 1.4 came out we excitedly upgraded, only to find out that we had an interesting interaction with the new stack moving and the escape analysis. We were passing a reference to a local variable into Cgo, and who knows what happened with the escape analysis but for some reason Go kept the variable stack-allocated. Anyway, unsurprisingly everything started failing miserably whenever the stack address changed. I'd be happy to share more details if this is a bug, but given the dire warnings about passing Go allocated memory into C, we just expected this sort of thing was expected and would continue to break in newer Go versions. Hence, we're still on 1.2 until we can do a big audit of all of our Cgo to eliminate this case as much as possible.

-JT

[kane....@gmail.com]

It's been a while since last activity in this thread. What's the situation nowadays?

[Ian Lance Taylor]

On Sun, Sep 4, 2016 at 1:19 AM, <kane....@gmail.com> wrote:
> It's been a while since last activity in this thread. What's the situation
> nowadays?

The rules for cgo pointer passing have been decided and are documented
at https://golang.org/cmd/cgo .

Ian