Mysterious RSS memory spike

[Manish R Jain]

Hey Gophers,

I'm puzzled by a mysterious RSS memory spike in my Go program, when all memory allocations are happening via Cgo. I assert that there are no memory leaks in the program. And have written another C program with similar logic which does NOT show RSS memory spiking. So, I suspect this is something to do with Go memory.

Program:

https://github.com/dgraph-io/ristretto/pull/186

This PR creates a Go memtest program, which does this:

- Uses z.Calloc and z.Free to allocate Go struct (S) and a byte slice inside it. All allocations are happening in Cgo, and being type casted into Go. No allocations are happening in Go (except a 32 MB fill slice).

- z.NumAllocBytes is tracking memory allocated and freed by these calls.

- Increases memory usage to 16 GB (as reported by z.NumAllocBytes).

- Decreases it back to 1 GB.

- Repeats this cycle.

- On Ctrl+C, it deallocates everything and asserts that Cgo memory allocated is zero.

I was concerned about memory fragmentation, so created a very similar C program which does the same thing (memtestc).

Please feel free to run either of the Go or C programs. They should compile and run easily.

Behavior:

Run the program with: `go build . && ./memtest` . Go pprof heap shows 32 MB used, to account for the fill slice. However, RSS reported keeps roughly increasing every cycle.

I'm using Go 1.14.4 and on it, RSS jumps to 22GB after a few cycles. memtestc (C equivalent, compiled with gcc) does not show this behavior. The RSS goes down to 1GB-ish every cycle.

Any clues why the RSS is much higher than expected in Go and keeps climbing in Go?

—

Manish

Founder, https://dgraph.io

[Mike Spreitzer]

BTW, how are you measuring RSS? That is a trick all by itself! See
https://groups.google.com/g/golang-nuts/c/LsOYrYc_Occ/m/LbjLAsL6BwAJ

Regards,
Mike

golan...@googlegroups.com wrote on 08/23/2020 11:04:47 AM:

> ?
> Manish
> Founder, https://dgraph.io
> [image removed]
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/
> d/msgid/golang-nuts/a592db0482f1843c694a9486379a47dd%40frontapp.com.

[Vladimir Varankin]

Hey,

I haven't looked deep but I recall there had been a note about runtime change in Go 1.13's https://golang.org/doc/go1.13#runtime That is

> The runtime is now more aggressive at returning memory to the operating system to make it available to co-tenant applications [..] However, on many OSes, including Linux, the OS itself reclaims memory lazily, so process RSS will not decrease until the system is under memory pressure. 

Could that be the behaviour you observe (although, since you don't see the same in C implementation, I might be confused, sorry in advance).

[jake...@gmail.com]

If it helps any, I do not see this in Windows 10. After 5,000 iterations the "low memory" mark, as reported by windows, is still just 1,101MB.

[Miha Vrhovnik]

Hey, Vlad,

it's not that. we have a similar or rather the exactly the same problem... where the OS doesn't claim the memory back.. this is also annoying to the point we had to disable memory under pressure notifications for that specific host.

Also valgrind doesn't report any leaks... simulating the same with C app the memory is returned to the OS almost immediately.

Fortunately for us this is in the worker processes where we can restart them after about 1000 iterations.

BR,

Miha

[Manish R Jain]

Thanks to Kale in darkarts Slack channel, we were able to identify the issue. The default C Calloc has major memory fragmentation issues due to Go's multi-threaded execution (or so is our theory). Switching Cgo to use jemalloc instead fixed the issue. The RSS now goes down to (close to) 1GB on every cycle. Anyone interested can see the linked PR for code updates.

I'll write a blog post about this soon-ish.

—

Manish

Founder, https://dgraph.io