Lots of retained free spans

[Eric Hubbard]

I've been trying to root cause an OOM condition.  My process is running within a cgroup with a 4gb limit.  Occasionally that limit gets hit.  I added a cgroup listener to watch for memory usage and create a pprof and core dump.

The pprof shows just a few hundred megs of "in-use" memory, however when I open the core dump with viewcore I'm seeing around 3.3gb of free spans that are being retained.  "kept for reuse by Go"..

I've got GOGC set to 50 -- but my understanding is that just controls when a GC kicks off -- not when memory will be returned to the OS.   

Is there some sort of behavior a program can do to create this type of situation?  or a way to give a hint to the GC  .. "hey... give it back!" :)   

Thanks for any ideas or thoughts!

[Brian Candler]

What go version are you compiling with?

go prior to 1.16 would hold onto memory such that the OS would only reclaim it when under memory pressure; it could be made to free memory eagerly using GODEBUG=madvdontneed=1.  This was made the default in go 1.16.

https://discuss.dgraph.io/t/benchmarks-using-godebug-madvdontneed-environment-variable/7322
https://go.dev/doc/go1.16#runtime

[Eric Hubbard]

v1.16.4

[Eric Hubbard]

How does go know about memory pressure?  maybe that aspect is broken in my environment... within a cgroup..etc..

On Saturday, 29 January 2022 at 02:16:56 UTC-8 Brian Candler wrote:

[Brian Candler]

Go doesn't know about memory pressure, but rather it informs the OS about pages it no longer needs, and the OS deals with memory pressure. See the madvise(2) manpage and the go1.12 release notes.

If I understand correctly, MADV_DONTNEED tells the OS to discard immediately.  Between go1.12 and go1.15 it used MADV_FREE, where the expected OS behaviour is that "the freeing could be delayed until memory pressure occurs"

[Gleb Petrovichev]

I'm having the exactly same problem now. Did you by any chance able to figure it out? I'm compiling with go1.19.8 though. Thanks

[Robert Engels]

Go GC does not use a compacting/copying collector so depending on your allocation patterns you could see this behavior. But you’re stating > 70% free spade available. 

I.e It may be that these are holes that cannot be released to the OS. 

On May 29, 2023, at 10:08 AM, Gleb Petrovichev <petrovic...@gmail.com> wrote:

I'm having the exactly same problem now. Did you by any chance able to figure it out? I'm compiling with go1.19.8 though. Thanks

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e7c467fa-d78a-4b1c-b14a-18e0d028810bn%40googlegroups.com.