CGO - Passing pointer to C

[Robert Johnstone]

Hello,

The section on passing pointer to C in the cgo documentation is quite clear that letting C hold pointer to Go memory is not allowed.  I'd like to better understand the limitation.  Frankly, because of the architecture of a system I'm working on, the inability to transport pointers is complicating matters.

1) The Go objects in question are certainly on the heap.

2) The references in C are not needed to maintain the lifetime of the Go objects.

My understanding is that the main reason for the restriction is that Go might move to a moving garbage collector.  At the moment, unless there is definite work on this, I'm inclined to break the rule and let the C code hold onto pointers to Go memory.  However, before committing to this approach, I'd like to make sure I haven't missed any other reasons for the prohibition.

Thanks.

[Ian Lance Taylor]

There is more background on this at
https://github.com/golang/proposal/blob/master/design/12416-cgo-pointers.md
.

The Go compiler compiles code to use a write barrier when storing
pointers to memory. The C compiler obviously does not. When storing
a Go pointer in C code, no write barrier is executed. This can cause
the garbage collector to be confused about whether the pointer is
reachable. In particular, it is absolutely critical that C code never
store a Go pointer into Go memory.

Still, while you have to be careful and I recommend against it, you
may be able to get away with letting C code hold onto Go pointers
across calls, in C memory. The Go values must be in the heap, not the
stack. The Go values must be kept alive by existing Go pointers.

But it's risky and I personally would never do it. Much better, if
possible, to use a handle. Store the Go pointer in a map on the Go
side, and pass the map key, an integer, to the C side. Then the C
side can pass the integer back, and Go code look it up in the map to
fetch the pointer.

Ian

[Robert Johnstone]

Hello,

Thanks for the quick reply.  I had not considered the write barriers, but if the Go objects are "live" longer than the held in C, it would work.

I definitely agree that there are risks associated with this approach.  We are giving up some of the safety of Go.  Unfortunately, we are using cgo for more than some computation, so we have live objects in C as well, and so we cannot completely escape the manual memory management required.

What is the state of plans for a moving garbage collector?  This would definitely wreck havoc if any pointers to Go memory were held in C.

Thank-you,

Robert

On Tuesday, 3 December 2019 22:39:35 UTC-5, Ian Lance Taylor wrote:

[Ian Lance Taylor]

On Wed, Dec 4, 2019 at 6:48 AM Robert Johnstone <r.w.jo...@gmail.com> wrote:
>
> Thanks for the quick reply. I had not considered the write barriers, but if the Go objects are "live" longer than the held in C, it would work.

The write barriers do not look only at the pointer being stored, they
also look at the contents of memory being stored into. That is why C
code must never store a Go pointer into Go memory.

> I definitely agree that there are risks associated with this approach. We are giving up some of the safety of Go. Unfortunately, we are using cgo for more than some computation, so we have live objects in C as well, and so we cannot completely escape the manual memory management required.
>
> What is the state of plans for a moving garbage collector? This would definitely wreck havoc if any pointers to Go memory were held in C.

There are no current plans for a moving garbage collector.

I cannot promise that no other changes will break this approach.
Obviously we won't consider bug reports for code that breaks the cgo
rules.

Ian

[Rick Hudson]

Breaking the Go CGO pointer rules comes up periodically and the rules 

have not changed. Applications have lived with the rules simply 

because breaking them results in revisiting the application code 

every time a new Go release comes out. Did the compiler improve and 

some object is now allocated on the stack instead of the heap? Did the 

runtime borrow some MESH [1] virtual memory page fragmentation 

techniques but improve them for Go by updating pointers to reducing 

TLB pressure? Is there value in moving objects from NVRAM to DRAM 

and updating pointers? And so forth and so on. Nobody knows if any of 

this will ever happen but the Go CGO pointer rules leave open the possibility.

[1] Bobby Powers, David Tench, Emery D. Berger, and Andrew McGregor. 2019. 

Mesh: Compacting Memory Management for C/C++ Applications. In 

Proceedings of the 40th ACM SIGPLAN Conference on Programming Language 

Design and Implementation (PLDI ’19), June 22ś26, 2019, Phoenix, AZ, USA. 

ACM, New York, NY, USA, 14 pages. https://doi.org/10.1145/3314221.3314582

On Wednesday, December 4, 2019 at 11:14:17 AM UTC-5, Ian Lance Taylor wrote:

[Michael Jones]

Speaking for the Rick Hudson Fan Club, thank you for your wonderful, amazing work on the Go GC...

https://blog.golang.org/ismmkeynote

...and for this reminder that we don't want programming practice to preclude continuing innovations.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/23885bd0-4802-40c7-b4c9-52541e92029f%40googlegroups.com.

--

Michael T. Jones
michae...@gmail.com