cgo keep consuming memory for SysStack until OOM

[Renat Idrisov]

Hi All,

we use cgo on Linux, because of mips bytecode that is not supported by main go compiler.

golang version is 1.16.5, latest release didn't tried. Writing here to make sure what we see is a bug.

The application is running for a few days, alive allocations are less than 100MB:

# Alloc = 36761056

# TotalAlloc = 122637932112

however this number keep growing until it got killed by OOM:

# Sys = 852037904

# Stack = 0 / 637796352

the number of OS threads is not high, all counts look very modest:

Count Profile

1810 allocs

0 block

0 cmdline

18 goroutine

1810 heap

0 mutex

0 profile

8 threadcreate

0 trace

what can I do to prevent it from consuming that much stack and crashing?

complete memstats below:

# runtime.MemStats

# Alloc = 36761056

# TotalAlloc = 122637932112

# Sys = 852037904

# Lookups = 0

# Mallocs = 359316187

# Frees = 359262165

# HeapAlloc = 36761056

# HeapSys = 201326592

# HeapIdle = 161890304

# HeapInuse = 39436288

# HeapReleased = 137854976

# HeapObjects = 54022

# Stack = 0 / 637796352

# MSpan = 294984 / 376832

# MCache = 5016 / 16384

# BuckHashSys = 3459520

# GCSys = 8067352

# OtherSys = 994872

# NextGC = 71737968

# LastGC = 1633401481045016063

Thanks!

[Renat Idrisov]

in addition to the original question, I have found a way to get the following memstats in a few seconds:

# runtime.MemStats

# Alloc = 22267160

# TotalAlloc = 42133760

# Sys = 8852177152

# Lookups = 0

# Mallocs = 1041166

# Frees = 805749

# HeapAlloc = 22267160

# HeapSys = 67108864

# HeapIdle = 42459136

# HeapInuse = 24649728

# HeapReleased = 41943040

# HeapObjects = 235417

# Stack = 0 / 8778907648

# MSpan = 384064 / 393216

# MCache = 5016 / 16384

# BuckHashSys = 1452992

# GCSys = 3676424

# OtherSys = 621624

# NextGC = 27397520

# LastGC = 1627069277209120125

7 threadcreate

7 goroutine

Total allocations of golang program are about 42MB, but go runtime takes 8GB for in OS thread stacks.

What am I doing wrong?

Thanks!

[Jan Mercl]

On Wed, Oct 6, 2021 at 5:43 PM Renat Idrisov <parsi...@gmail.com> wrote:

> Total allocations of golang program are about 42MB, but go runtime takes 8GB for in OS thread stacks.
> What am I doing wrong?

If you're on 64 bit Linux, chances are the default OS thread stack
size is 8 MB. In that case your program uses ~1000 OS threads. That
might be unexpected or it might be caused by some bug. So you should
probably focus first on finding out why your program uses such
many/too many OS threads, provided this hypothesis is somehow
connected to reality.

[Renat Idrisov]

Thanks Jan,

that is a good explanation since I do use a lots of gorotines,

but I supposed they are not mapped to OS threads that frequently.

Correct me if I am wrong, as far as I know there is no way to recycle them.

Is it correct that any IO to socket or file makes gorotine an OS thread?

What about mutexes and sleeps?

I can rethink the architecture to do all IO from dedicated gorotines,

but need to confirm the list of "forbidden" operations.

Thanks again!

[Jan Mercl]

On Wed, 6 Oct 2021, 18:38 Renat Idrisov, <parsi...@gmail.com> wrote:

... I do use a lots of gorotines,

but I supposed they are not mapped to OS threads that frequently.

The runtime attempts to limit the number of OS threads to the value of GOMAXPROCS. However, some threads do not count against that budget, namely CGo calls that have to execute in a dedicated OS thread because a goroutine stack might be not big enough for running C code in the general case. If there are many goroutines doing CGo calls it may explain your observations.

Correct me if I am wrong, as far as I know there is no way to recycle them.

I'm not sure on this. I think OS threads are not destroyed by the runtime, but parked and reused when a new one is needed.

Is it correct that any IO to socket or file makes gorotine an OS thread?

I think some syscalls spawn a thread when they do not return within some time limit.

What about mutexes and sleeps?

Those should be safe AFAICT. But I know very little about the runtime above some basic concepts.

I can rethink the architecture to do all IO from dedicated gorotines,

but need to confirm the list of "forbidden" operations.

I hope some expert will chim in and provide an authoritative answer to this.