Investigating suspected memory leak based on StackInUse or HeapSys stats

[Shlomi Amit]

Hi.

I’m trying to find a memory leak in my application.
I’ve added some runtime memory stats to the logs (Heap & Stack stats.. go routine and heapObject counts).
I can see that from time to time m.StackInuse is increasing without any obvious reason from the application perspective (At least not one that I can find yet).
Here some additional runtime mem stats (Taken at the same time):
StackInUse 56MB
HeapSys: 147MB.
HeapAlloc: 97MB

While StackInUse and HeapSys seems to increase from time to time, HeapAlloc stays about the same during 6 hours while the application is running.
There seems to be a bit of correlation between StackInUse and HeapSys, but overtime StackInUse increase more than HeapSys. (Is HeapSize includes StackInUse?)

In addition to adding runtime memory logs, I’m also creating periodic heap profile dumps.
My problem is that when analyzing the heap with pprof, it gives me no clue why StackInUse is so high.
The pprof inuse_space shows:
“Showing nodes accounting for 55.31MB, 93.25% of 59.31MB total”
What this total MB represent? It doesn’t seem to match HeapAlloc, HeapSys or StackInUse.
Does pprof heap profile even include StackInUse?

I really need to understand where the leak is coming from, but after looking in many places, the memory stats are still not clear to me, and neither what memory stat pprof heap profile really represent.
Note that I’m also logging HeapObjects count and I don’t see any leak there… It’s just StackInUse increasing from time to time (As it seems, it's always double itself... ~8MB->16MB->32), and HeapSys.

Note that I’m also using Cgo, but my understanding is Cgo memory allocations will not be reflected by the runtime memory stats. Is this correct and I assume if runtime memory stats are increasing this is defiantly because of go code and not C code?
I hope I was clear, but added a screenshot for the different memory stats.
Marked in red the point in times that stackInUse increase. (My current understanding which might be wrong, is that stackInUse is not included in HeapSys, this is why they are stacked in the graph).

I know my write is a bit messed up and you might not really be sure what's being asked, so in if I'll try to summarize:

1. What stackInUse represents? Is it part of HeapSys?
2. What HeapSys represents? (Both it and StackInUse are way more high than heapAlloc)
3. Why pprof inuse_space doesn't seem to have any notion of stuff which was allocated by StackInUse or HeapSys? It seems to pretty much corelate with HeapAlloc.
4. Since I'm also using Cgo, I do understand C memory allocation will not be reflected in pprof, perhaps they are being reflected by StackInUse or HeapInUse?
   Considering StackInUse seems to double itself in size when it increase, it seems like some kind of memory pre-allocation to have enough headroom than needed, not something which I expect to happen by C code.
5. Most importantly, with the below memory stats being increased, what's the approach to investigate?
6. Currently I'm using docker, and sometimes get OOM. What's best memory stats which will the most close to describe container RSS memory in use? (Specifically, my docker container was limited to 300MiB)

I know these are many question, but I did try and go over the runtime memory stats documentation, and I'm still very confused, especially not after the actual memory stats I'm seeing.
BTW, I'm using go 1.17.5

[robert engels]

Are you certain the number of Go routines is not increasing - each Go routine requires stack. See https://tpaschalis.github.io/goroutines-size/

<Untitled.jpg>

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com.
<Untitled.jpg>

[Shlomi Amit]

Yes. I already monitoring the runtime stat of number of go routines (Can be seen in the screenshot as well) and it's not increasing.

[robert engels]

Are you certain your CGo code isn’t creating threads?

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAbwE0McE6OULqOyrjHtw6tp493%3DLeDm624_tck71RDfkN%2BOEQ%40mail.gmail.com.

[Shlomi Amit]

I've already checked process threads using ps command, and threads are not leaking.

Still, it is more than possible that there are memory leaks within my Cgo code, or gstreamer as well (Which I'm running with Cgo as well)

But before I'm starting to chase this path, I would like to be sure heapSys or StackInUse stats are also including Cgo memory allocations, because my understanding was that Go runtime is not reflecting memory allocations done from C code.

I've aleady found Go memory leak before with pprof (Due to leak seen in heapObjects count) and another leak in goRoutine and fix them... But now I'm with the mystery of pprof showing low memory usage but stackInUse & HeapSys are increasing and are not reflected in pprof.

So - Are HeapSys or Stack In use includes memory allocations done in C code? If so, I'll move on and start chasing C code (Possibly using Valgrind? Can I use it to profile my Go application with Cgo?)

[robert engels]

Neither of those track C allocations - unless the C is calling back into Go.

You can review the low level usage in stack.go

Note that Go stack allocations are for the most part done in the heap, and are dynamically managed across Go routines.

Are you possibly doing some very deep recursions are times? Because the stacks will not shrink under certain conditions.

Since they are allocated on the heap, you may be encountering a situation where you are extended the stack to a degree that the GC cannot keep up 

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAAbwE0O6Wj2ngcGymiJhrwUxOJLQ_5TZbJgqWnqm_c_BZc8-bQ%40mail.gmail.com.

[Shlomi Amit]

Thank Robert.

My C code does call back Go code, but no recursion there.

If stack allocations are part of the heap, why it is not reflected in heapAlloc and pprof?

[robert engels]

Because they are tracked separately. See allocSpan() in mheap.go.

[Shlomi Amit]

So, I've configured pprof over http and downloaded the goroutines stack trace, apparently I did have an unintended endless recursion (1 additional function call every 5s) and I've fixed the issue.

stackInUse is now normal.

If I understand correctly, now it also makes sense why stackInUse didn't increase slowly (every 5s), but doubled itself after sometimes and each time the duration before the doubled increased was also doubled.

e.g. 2h on ~8MB before doubled to 16MB, 4h on 16MB before doubled to 32MB, 8h on 32MB before doubled to 64MB

According to this thread:

https://stackoverflow.com/questions/25317838/how-to-identify-the-stack-size-of-goroutine
As of go 1.3 it also uses an exponentially increasing stack size....

Thanks Robert for the help.