terminating goruntime internal threads

[Pavan]

Hi, 

how do we terminate the go generated internal OS threads.  I am debugging an issue , where  RES(memory resdent size)  size of. a process increases as goroutines used increases. 

Iam trying to reduce the threads footprint contribution of process RES , hence.  terminate the additional threads once go routines are completed.  

Is there an explicit call, I can make to terminate these processes. 

For the OS threads because of C functions calls, I am doing LockOSThread and skipping unlock, so that they terminate as goroutine terminates. 

I see some discussion of this here: 

https://github.com/golang/go/issues/14592

but could not find if an explicit call was introduced for application to call terminate. 

Regards,

[Ian Lance Taylor]

The trick using LockOSThread and returning from the function is
currently the only way to remove an existing thread. In general the
Go runtime assumes that if you needed a thread once, you might need it
again at some point.

If your memory usage increases without bound, it's fairly unlikely
that the problem is the number of threads. It's much more likely that
you have a memory leak in your Go program. Use the heap profiler.

Ian

[Pavan]

Thanks Ian. I start some 20 go routines and each routine involves calling C functions and finally all go routines terminate. since each goroutine uses LockOSThread at the begining, all the pthreads spawned as part of goroutines exit too. 

Now i see around 34 goroutines, few of them have stack size of 7 MB as seen in /proc/pid/smaps under Private_Dirty.  If i limit the stack size, looks like memory is instead taken from heap. I am trying to reproduce with simple example. Any thoughts/comments, Please share.

Regards,

[Ian Lance Taylor]

On Wed, May 6, 2020 at 10:02 AM Pavan <sudars...@gmail.com> wrote:
>
> Thanks Ian. I start some 20 go routines and each routine involves calling C functions and finally all go routines terminate. since each goroutine uses LockOSThread at the begining, all the pthreads spawned as part of goroutines exit too.
>
> Now i see around 34 goroutines, few of them have stack size of 7 MB as seen in /proc/pid/smaps under Private_Dirty. If i limit the stack size, looks like memory is instead taken from heap. I am trying to reproduce with simple example. Any thoughts/comments, Please share.

Goroutine stacks always come from the Go heap.

In a program that uses cgo, like yours, each thread will have a stack
allocated when the thread is created. That stack is used when calling
C functions. Note that this is each thread, not each goroutine. Most
programs have many more goroutines than threads (but this is obviously
not true for goroutines that call LockOSThread).

Why do you call LockOSThread, and why do you let the goroutines exit
with the thread locked? That is going to make your program run slower
and likely use more memory. It makes sense if the C code uses
thread-local storage or if the C code modifies the thread in some way
that makes it unusable by future goroutines. But without specific
reasons, it seems like a bad choice.

Ian

> On Saturday, April 25, 2020 at 6:51:37 AM UTC+5:30, Ian Lance Taylor wrote:
>>
>> On Fri, Apr 24, 2020 at 3:39 AM Pavan <sudars...@gmail.com> wrote:
>> >
>> > how do we terminate the go generated internal OS threads. I am debugging an issue , where RES(memory resdent size) size of. a process increases as goroutines used increases.
>> > Iam trying to reduce the threads footprint contribution of process RES , hence. terminate the additional threads once go routines are completed.
>> >
>> > Is there an explicit call, I can make to terminate these processes.
>> >
>> > For the OS threads because of C functions calls, I am doing LockOSThread and skipping unlock, so that they terminate as goroutine terminates.
>> >
>> > I see some discussion of this here:
>> > https://github.com/golang/go/issues/14592
>> > but could not find if an explicit call was introduced for application to call terminate.
>>
>> The trick using LockOSThread and returning from the function is
>> currently the only way to remove an existing thread. In general the
>> Go runtime assumes that if you needed a thread once, you might need it
>> again at some point.
>>
>> If your memory usage increases without bound, it's fairly unlikely
>> that the problem is the number of threads. It's much more likely that
>> you have a memory leak in your Go program. Use the heap profiler.
>>
>> Ian
>

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/94003c4b-9e29-4a41-8200-8b1cec062e19%40googlegroups.com.

[Pavan]

correction just to clarify in my previous mail : please read it as "around 34 go internal pthreads"  instead of "around 34 goroutines"

Ian, the go internal pthreads left out after termination of all the goroutines have the data (few strings allocated in application) as it is. 

I am trying to measure RSS of my process after all goroutines terminate.  It remains high, some of which seems to be part of go internal pthreads stack size. so i was thinking if there is a way to clean up these threads and see if RSS of application reduces. 

I am doing some more tests to confirm, if this stack size increase is due to memory leak bug in my go application .  

Yes , I was using LockOSthread because the C library we are using has some thread local storage used. 

Regards,

> To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.