CGO Threads and Memory Not Being Released in Go

[David Bell]

Hi everyone,

I'm relatively new to Go and even newer to CGO, so I’d really appreciate any guidance on an issue I’ve been facing.

Problem Overview

I noticed that when using CGO, my application's memory usage keeps increasing, and threads do not seem to be properly cleaned up.
The Go runtime (pprof) does not indicate any leaks, but when I monitor the process using Activity Monitor, I see a growing number of threads and increasing memory consumption.

Observations

• Memory usage keeps increasing over time when using CGO, even after forcing garbage collection.
• Threads spawned for CGO calls appear to not be released, causing a large number of lingering threads.
• The issue is not present when using pure Go time.Sleep() instead of the CGO function.

Reproducible Example

I created a minimal program that reproduces the issue. The following CGO-based code keeps allocating memory and does not free threads properly:

package main

import (
"fmt"
"runtime"
"runtime/debug"
"sync"
"time"
)

/*
#include <unistd.h>
void cgoSleep() {
  sleep(1);
}
*/
import "C"

func main() {
start := time.Now()

var wg sync.WaitGroup
for i := 0; i < 5000; i++ {
wg.Add(1)
go func() {
defer wg.Done()
C.cgoSleep()
}()
}
wg.Wait()

end := time.Now()

// Force GC and free OS memory
runtime.GC()
debug.FreeOSMemory()
time.Sleep(10 * time.Second)

var m runtime.MemStats
runtime.ReadMemStats(&m)

fmt.Printf("Alloc = %v MiB", m.Alloc/1024/1024)
fmt.Printf("\tTotalAlloc = %v MiB", m.TotalAlloc/1024/1024)
fmt.Printf("\tSys = %v MiB", m.Sys/1024/1024)
fmt.Printf("\tNumGC = %v\n", m.NumGC)
fmt.Printf("Total time: %v\n", end.Sub(start))

select {}
}

Expected Behavior

• The memory usage should not continue rising indefinitely.
• Threads should be properly cleaned up when they finish executing.
• The behavior should be similar to the following pure Go equivalent, which does not exhibit the issue:

Actual Results With CGO (cgoSleep()):

• Memory Usage: 296 MB
• Threads: 5,003
• System Memory (Sys from runtime.MemStats): 205 MB
  
  
  

With Pure Go (time.Sleep()):

• Memory Usage: 14 MB
• Threads: 14
• System Memory (Sys from runtime.MemStats): 24 MB

Additional Attempt

I tried forcing thread cleanup using runtime.LockOSThread() and runtime.Goexit(), but while the number of threads decreases, memory is still never fully released:

go func() { runtime.LockOSThread() defer wg.Done() C.cgoSleep() runtime.Goexit() }() Questions

1. Why is memory increasing indefinitely with CGO?
2. Why are threads not getting properly cleaned up after CGO calls?
3. Is there a way to force the Go runtime to reclaim memory allocated for CGO threads?
4. Is there a better approach to handling CGO calls that spawn short-lived threads?
5. Would using runtime.UnlockOSThread() help in this case, or is this purely a CGO threading issue?
6. Is there a way to track down where the memory is being held? Since pprof does not show high memory usage, what other tools can I use?

Go Version 
go1.23.5 darwin/arm64

[robert engels]

This is going to create 5000 OS threads. The loop runs really quickly, and since CGO cannot use Go routines, you quickly allocate 5000 OS threads.

See https://github.com/golang/go/issues/14592

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/5e8c1622-e6f0-47a8-a869-e78797800fb5n%40googlegroups.com.

[robert engels]

One way you can address this is to put a semaphore on the Go side around the C call, so you ensure only so many C calls are made simultaneously.

On Mar 4, 2025, at 1:26 PM, David Bell <davidbe...@gmail.com> wrote:

[robert engels]

Or at the github issue points out, if the thread will exit, LockOSThread my work.

[David Bell]

Thanks for the response Robert, 

I tried to put a semaphore to control the max amount CGO calls simultaneously,
similar to what they did here 
https://github.com/golang/go/blob/master/src/net/net.go#L794-L818

i noticed it does help somewhat, but memory consumption keeps going up uncontrollably.



I tried to do LockOSThread and use GoExit so the thread will close. 
it indeed closes the thread but the memory is never released back to the os.

is there a way i can force it to release the resources back to the os when the thread is closed?

ב-יום שלישי, 4 במרץ 2025 בשעה 22:32:45 UTC+2, robert engels כתב/ה:

[Ian Lance Taylor]

On Tue, Mar 4, 2025 at 11:32 AM David Bell <davidbe...@gmail.com> wrote:
>
> Why is memory increasing indefinitely with CGO?

Because when the Go scheduler creates threads, it keeps them around,
on the theory that if they were needed once, they will be needed
again. Certainly we wouldn't want all threads to exit as soon as they
no longer needed, as for most programs that would waste time exiting
and recreating threads. Your program creates a lot of threads.
Developing a better approach for this is https://go.dev/issue/14592.

It's not a problem in practice for most programs, because most
programs do not run 5000 executions of C code in parallel. But as
you've seen it's certainly possible to write a program that behaves
badly in this regard.

> Why are threads not getting properly cleaned up after CGO calls?

Same answer.

> Is there a way to force the Go runtime to reclaim memory allocated for CGO threads?

Not really, no. As others have pointed out you can use
runtime.LockOSThread in a goroutine, which will cause the thread used
for that goroutine to exit when the goroutine exits. But it's not a
very elegant solutoin.

> Is there a better approach to handling CGO calls that spawn short-lived threads?

There are various approaches, but I think they all start with: why are
you doing this? Your example program demonstrates the problem but is
clearly pointless, so what does your real program do?

> Would using runtime.UnlockOSThread() help in this case, or is this purely a CGO threading issue?

runtime.UnlockOSThread won't do anything, but as noted
runtime.LockOSThread can change the behavior.

> Is there a way to track down where the memory is being held? Since pprof does not show high memory usage, what other tools can I use?

It sounds like most of the memory is being held by thread stacks
created by the C library. Unfortunately, as far as I know, Go doesn't
have a good way to track that.

Ian

[robert engels]

How many threads are you limiting it to? The OS thread stacks can be large. A basic premise of Go is to not use threads, and use Go routines - harder with CGO - but you often need to think about rearchitecting to make the C layer a “service handler” - like any other service - and then limit the number of requests in flight.

When I limit it to 64 threads on my Mac, I see about 16mb memory being used.

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/9a40383e-5b47-4f2b-bdea-a50946381650n%40googlegroups.com.

[David Bell]

Hi Ian and Robert,

First, thank you both for taking the time to answer and provide insights! I really appreciate it.

To clarify, my real program is a long-lived process that receives network traffic from C++ code and processes it in Go.
The CGO calls in my program seemed to be the likely culprit behind the increasing memory usage I observed.

I tried adding runtime.LockOSThread() around the CGO calls and used a semaphore to limit concurrency.
This change has significantly improved the issue, as the memory growth is now much more controlled.

However, there’s something I still don’t fully understand. In my toy example, where I spawn 5000 CGO calls, locking the thread indeed causes Go to properly clean up the threads, and I end up with the correct number of threads. But despite that, memory usage still remains around 200MB.

If locking the thread ensures proper cleanup, why does memory usage remain high in the toy example, while in my real program, it seems to help? Is there some underlying mechanism in Go’s thread management or memory allocator that could explain this difference?

Thanks again for your help!

[David Bell]

I wanted to add some clarification on my misunderstanding and the approaches I tried.

I tested two different strategies to handle the CGO calls:

1. Worker Pool Approach (50 workers handling CGO calls)

   • I set up a worker pool with 50 goroutines, each making CGO calls in a controlled manner.
   • Surprisingly, this did not resolve the issue—memory usage still kept increasing.

2. Locking OS Threads with a Semaphore (Up to 500 concurrent CGO calls)

   • Instead of using a worker pool, I tried having each CGO call run in its own goroutine but with runtime.LockOSThread().
   • I also used a semaphore to ensure that at most 500 CGO calls could run concurrently.
   • This approach did resolve the issue—memory usage stopped growing indefinitely.

What I don’t quite understand is why the second approach works while the first one doesn’t.
If the worker pool limits concurrency to 50 workers, why does it still cause memory growth, while 500 locked threads behave better? Is it because of how Go manages OS threads internally?

Appreciate your thoughts on this!

[robert engels]

I suspect your implementation is incorrect in some way, or you have a memory leak of another sort.

When I used your original code, the memory usage according to the activity monitor was > 100 mb.

When I changed it to use a semaphore to limit the concurrency to 64 threads, it was about 16mb.

You also have to be aware that Go does not have a compacting collector, so depending on your allocation pattern it may be nearly impossible to release the physical pages back to the OS. It can release physical pages back to the OS if the garbage collector can determine that nothing in the address range is being used.

So bottom line, you have to be careful about which memory statistics you use to “measure memory” with Go.

This has a lot of great information here https://tip.golang.org/doc/gc-guide#A_note_about_virtual_memory

To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/82290274-54d4-41c5-8c9f-4d881e6b474dn%40googlegroups.com.