Automatically releasing memory resources allocated in cgo?

[Tom Payne]

cgo is often used to provide bindings to C libraries. Any memory allocated in the C library is not visible to Go, so Go does not have an accurate view of the program's memory usage and does not run the garbage collector or any finalizers often enough. Consequently, memory usage for a Go server that uses cgo heavily can grow very large, with Go itself being utterly unaware of it.

If the C functions allocate memory, historically you could set a finalizer to free the memory sometime after there are no remaining references to it in Go, as, for example, described in this blog post. However, the current Go docs on runtime.SetFinalizer state:

> There is no guarantee that finalizers will run before a program exits, so typically they are useful only for releasing non-memory resources associated with an object during a long-running program.

Are there any other ways to automatically release memory resources associated with an object? Or telling Go's memory manager that the small object it sees in the Go world is but the tip of an iceberg of memory allocated in the C world and therefore should be finalized?

Not-good options include:

- Requiring the programmer to make explicit calls to free the memory resources when they believe the object is no longer needed, but this takes us back to the painful world of C's manual memory management and is easy to get wrong.

- Padding Go objects associated with C memory resources with large, unused fields (e.g. an [1024]byte) in the hope that the Go garbage collector will be more likely to finalize and free them when they are unused.

- Avoiding cgo in any server code.

Are there any good options?

Cheers,

Tom

[Ian Lance Taylor]

If you are using C code, you are using C's memory resource policy. So
the best approach is to use explicit calls to free. I agree that that
takes you back to the painful world of C's manual memory management,
but, after all, if you are using C then you can't pretend that you are
not using C.

That said, although Go is careful not to promise that it will actually
execute finalizers, in practice it does run them. Padding the Go
object won't make any difference as to when a finalizer is run. It
will be run in the next full GC cycle after the Go value is no longer
referenced. If there are times when the program knows that there is a
lot of C memory that is no longer required, it can help by calling
runtime.GC itself.

In practice I think the best approach is a hybrid: free the C memory
explicitly, but also add a finalizer. In the finalizer, free the C
memory, and log the fact that you are doing so, along with any
information that will help identify where the C memory was allocated.
Then periodically check your logs for cases where the finalizer ran,
and add the necessary calls to explicitly free the C memory manually.
Of course, when you free the C memory manually, don't forget to clear
the finalizer.

Ian

[Jake Montgomery]

Ian's advice seems sound. But there is one other option to consider. When practical, I like to simply copy C allocated memory to Go allocated memory in the wrapper for the cgo call, and free the C memory immediately. This removes the insanity of keeping track of C memory in Go. Obviously there are myriad reasons why this might not be possible, or might impose to great a penalty. But where it is possible, it makes life easy again.

[Tom Payne]

Thank you very much for the fast, clear, and detailed answer :)

[Berkant Ay]

Hi jake could you give me an example how you achieve that. Because I did the same approach but still faults. However runtime.GC() works properly. But im not into using it.

```

bytes := C.GoBytes(unsafe.Pointer(reqProp.data), C.int(reqProp.size))

copiedDataSize := copy(reqPropDataGo, bytes)

C.free(unsafe.Pointer(C.CBytes(bytes)))

```

is this what you mean?

1 Kasım 2019 Cuma tarihinde saat 18:39:09 UTC+3 itibarıyla Jake Montgomery şunları yazdı: