Debugging memory leak when a GO process crashed

[mariappan balraj]

Hello Go experts,

I could able to find the solution to debug memory leaks of GO process, when it is running by using PPROF. Is it possible to collect the heap profile for debugging, when GO process crashed using core dump? or what is the recommended way to root cause the memory leak? Based on the need, enabling PPROF and debugging may not be possible in the production environment. Please help. 

Best Regards

Mariappan

[mariappan balraj]

Hello Go experts,

Can someone please help with this?

Best Regards

Mariappan

[Brian Candler]

I think you're not being clear on your problem you're trying to solve.

There are no "memory leaks" in Go in the traditional sense (omitting to free an allocation) because of the garbage collector. Therefore, if memory is not being freed in a pure Go program, it's because you are keeping a reference to it somewhere - e.g. in a global variable, directly or indirectly. Or it could be referenced from a local variable in a goroutine which isn't terminating - in that case I'd say that's a goroutine leak, not a memory leak.  Is that the sort of thing you're trying to debug?

Or, is the situation that you're using CGO and forgetting to free some allocations in your C code?

Or, I've seen some people complain that Go doesn't not release unused memory back to the operating system as quickly as they'd like or expect, showing as high RSS. If that's your issue then it can be discussed.

[mariappan balraj]

Hi Brian,

A Go program can crash because of Out of Memory. Yes it can be because of goroutine leak. Or it can have some reference to global variables, but it should be freed? How to debug this?

When the leak is happening while using CGO, is there any tool to debug this?

Best Regards

Mariappan

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/19a761a1-7707-4300-9128-93f22b9ee342n%40googlegroups.com.

[Robert Engels]

You can certainly have memory leaks in Go - due to undesired references retaining the object. What the op is asking for is a memory dump at OOM crash. 

If the dump shows 1M object X and the author believes there should only be 1 - you have a leak   

On Jan 12, 2023, at 4:01 AM, mariappan balraj <mariappa...@gmail.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAKKWi6SKZr9x7-YoJNLLp57815bzg40wjCsYSYxtd7SQ2VMEUA%40mail.gmail.com.

[Vladimir Varankin]

Hey Mariappan,

From my experience, there are several possible options:

I believe you can use GOTRACEBACK=crash env variable (or its equivalent in the runtime/debug package in the std) to get a coredump on the crash. See this old post from JBD [1], that explored this.

If you clearly observe, that the application "leaks" its memory (or it "leaks goroutines") via the metrics — simple by looking at the trends in the app's memory usage — you may collect several pprof files (heap and goroutines), with some interval. Then explore the diff between the two profiles with pprof tool's diff option. To help automating this, the continuous profiling became a fairly trendy thing in the recent years. I won't recommend the actual project I explained back then in the [3], but the overall idea still valid, and there are several big and small offerings for that nowadays.

Hope this will help.

[1]: https://rakyll.org/coredumps/

[2]: https://pkg.go.dev/runtime/debug#SetTraceback

[3]: https://medium.com/@tvii/continuous-profiling-and-go-6c0ab4d2504b