Deleting map entry from the top level of a nested map doesn't clean the underlying memory

[naveen...@gmail.com]

https://play.golang.org/p/e22ufH-T2M1 This is my sample data structure. package main import ( "fmt" ) type MicroChkpt struct { comprtype uint32 MicroChkptInfoMap map[uint32][]byte } type CallChkpt struct { FullChkptData []byte MicroChkptMap map[uint32]*MicroChkpt ckey uint32 comprtype uint32 AuditInProgress bool } var CallChkptMap map[uint32]*CallChkpt func main() { fmt.Println("Hello, playground") } So its a nested map structure, CallChkptMap->MicroChkptMap->MicroChkptInfoMap So i was expecting on deleting an entry from the top level map CallChkptMap, whole underlying memory used by nested maps would be reclaimed. Its not happening till all entries are removed from the top level map( then only i see memory dipping), map has ongoing insert and delete operations and grows pretty big. Any workarounds to reclaim the memory on deleting the specific entry please? Should i go to the nested maps first, set them to nil and then delete the entry from the top level map? Appreciate all your time and inputs.

[Ian Lance Taylor]

How exactly are you measuring whether the memory has been garbage collected?

Ian

[Naveen Kak]

I have my system up and running for let's say 10 hours or so where these operations on map ( continuous add/delete) keep happening. Memory keeps growing and goes very high. 

Eventually at end of test we clear all map entries, memory is reclaimed and is good. 

It's basically kind of load test. 

Appreciate your engagement. 

[Jake Montgomery]

Perhaps a small program that reproduces the issue would help.

I wrote a quick one: https://play.golang.org/p/hbnuZsEqUKV . However, after an hour of running I see no "leak". I am on "go1.14 windows/amd64". After  6,000,000 iterations and 4 TB total allocated The memory stats (HeapInuse and HeapAlloc) are still dancing around the same values they had at the start. And Windows still reports the same 11.9-12.1 MB of memory used that it did at the start.

On Monday, April 27, 2020 at 12:58:54 AM UTC-4, Naveen Kak wrote:

I have my system up and running for let's say 10 hours or so where these operations on map ( continuous add/delete) keep happening. Memory keeps growing and goes very high. 

Eventually at end of test we clear all map entries, memory is reclaimed and is good. 

It's basically kind of load test. 

Appreciate your engagement. 

On Mon, 27 Apr, 2020, 3:41 AM Ian Lance Taylor, <ia...@golang.org> wrote:

[Ian Lance Taylor]

On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak <naveen...@gmail.com> wrote:

I have my system up and running for let's say 10 hours or so where these operations on map ( continuous add/delete) keep happening. Memory keeps growing and goes very high. 

Eventually at end of test we clear all map entries, memory is reclaimed and is good. 

It's basically kind of load test. 

Appreciate your engagement. 

Thanks for the information, but you didn't really answer my question.  How exactly are you measuring whether the memory has been garbage collected?

[Naveen Kak]

Basically using the Top command at end of test.

Let me give a quick summary of the scenario.

Basically we have an application which keeps getting data on a TCP connection, we allocate a global slice to temporarily hold the data and then store in nested maps.

For every TCP data transaction, we store the data in a  global slice temporarily and then operate on the maps.

For a certain amount of data, i know what should be the total allocation, but when i check using top command at end of test its almost always double or slightly more.

if I call runtime.GC() on every tcp transaction, memory looks good and end of test it is as expected ( everything reclaimed)

Is it because the system is heavily loaded ( per sec we may be getting huge no of transactions) so garbage collector cant keep track of the objects to be freed.

As a test I even tried to block the delete operations on Map, just addition to maps on a smaller scale though, even then memory is almost double of what is expected.

only solution is calling runtime.GC() on every tcp transaction, then everything is fine.

I am really not sure whether the map deletions don't work properly or is it just the GC which is not able to work efficiently enough on its own.

Calling GC so often is an overhead and would cause CPU spikes.

[Kevin Chowski]

Guessing based on your latest description: are you aware that there is no partial slice collection in GC? That is:

 

bigSlice := make([]int, 1000*1000)

subSlice := bigSlice[0:1:1]

bigSlice = nil

runtime.GC()

// At this point, bigSlice is still allocated! It cannot be freed by the GC (in the current implementation)

useTheSlice(subSlice) 

You have to explicitly reallocate smaller slices (and copy over the data from the big slice into each smaller slice) in order for the GC to be able to collect each smaller slice individually.

 

As suggested earlier in the thread, if you can create and share a small, self-contained program which exhibits the behavior you're speaking of, someone on this forum is much more likely to be able to help you. Trying to guess how a memory leak is manifesting is just... guesswork without looking at full code. There are lots of reasons why memory may not be collected by the GC, including random bugs in your program :)

On Tuesday, April 28, 2020 at 2:00:55 PM UTC-6, Naveen Kak wrote:

Basically using the Top command at end of test.

Let me give a quick summary of the scenario.

Basically we have an application which keeps getting data on a TCP connection, we allocate a global slice to temporarily hold the data and then store in nested maps.

For every TCP data transaction, we store the data in a  global slice temporarily and then operate on the maps.

For a certain amount of data, i know what should be the total allocation, but when i check using top command at end of test its almost always double or slightly more.

if I call runtime.GC() on every tcp transaction, memory looks good and end of test it is as expected ( everything reclaimed)

Is it because the system is heavily loaded ( per sec we may be getting huge no of transactions) so garbage collector cant keep track of the objects to be freed.

As a test I even tried to block the delete operations on Map, just addition to maps on a smaller scale though, even then memory is almost double of what is expected.

only solution is calling runtime.GC() on every tcp transaction, then everything is fine.

I am really not sure whether the map deletions don't work properly or is it just the GC which is not able to work efficiently enough on its own.

Calling GC so often is an overhead and would cause CPU spikes.

On Tue, Apr 28, 2020 at 3:21 AM Ian Lance Taylor <ia...@golang.org> wrote:

[Robert Engels]

Also, it may just be that the runtime is better off allocating more and not doing a GC based on available memory and CPU usage. The “max heap” feature in development may help here. 

On Apr 28, 2020, at 3:18 PM, 'Kevin Chowski' via golang-nuts <golan...@googlegroups.com> wrote:



--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/f4b8f164-085c-4d3e-8f51-c16138c3ac83%40googlegroups.com.

[Ian Lance Taylor]

On Tue, Apr 28, 2020 at 1:00 PM Naveen Kak <naveen...@gmail.com> wrote:

Basically using the Top command at end of test.

The top command will show you the memory that the program has requested from the operating system and has not returned to the operating system.   The Go memory allocator works by requesting memory from the operating system as it needs it.  The Go garbage collector works by looking at that memory and marking it as available for future allocations by the Go memory allocator.  The Go garbage collector does not immediately return memory to the operating system.  That is because requesting from and returning to the operating system are relatively slow operations, and a program that has needed memory once is likely to need it again.

So the top program is not a good way to judge what the garbage collector is doing.  It is an OK way to judge the maximum memory use of the program, which will include memory that has been allocated and memory that has been garbage collected.

If a Go program runs for a while with excess memory, it will slowly return it to the operating system.  You can encourage that process by using runtime/debug.FreeOSMemory.

In general, though, if you want to examine the garbage collector, I recommend that you use runtime.ReadMemStats rather than top.

Ian

[Naveen Kak]

Hi Ian,

I explored a few things, calling debug.FreeOSMemory periodically. This does help, I see a definitely a change in the memory being returned to the OS ( looking at top o/p).

I also set the "GODEBUG=madvdonotneed=1", as per go documentation post 1.12 release, go release it uses "madvfree" option which is basically a lazy free to the OS.

This didn't surprisingly did not have any effect.

So one thing for sure that deleting map definitely doesn't have any bug, its the way GC is working, not releasing everything back to OS ( I think after running for 12 hours or so, if we leave the system idle i don't think memory gets released back to OS, GC probably thinks it will be required to ask for memory so holds on to it unless we call the debug.FreeOSMemory periodically).

What you think about using "sync pools" so that there are no frequent memory allocations/de-allocations?, haven't explored this much yet.

Another thing, calling debug.FreeOSMemory periodically does cause CPU spikes.

Regards

Naveen

  

[Naveen Kak]

Ian, 

Any thoughts on this? Appreciate a response. 

Thanks

Naveen

[Ian Lance Taylor]

On Thu, May 7, 2020 at 11:57 AM Naveen Kak <naveen...@gmail.com> wrote:
>
> Ian,
> Any thoughts on this? Appreciate a response.

I'm sorry, I'm not sure what you want a response on.

Using top is a fine way to see total memory usage of your application.
It's a terrible way to examine the Go garbage collector. Using
runtime.ReadMemStats will give you a better understanding of the
garbage collector.

Whether it makes sense to use sync.Pool depends on your application.
The sync.Pool documentation explains when it is useful.

Ian

[Urjit Singh Bhatia]

Hi Naveen,

Your expectations about the program immediately giving up memory on deleting an object are wrong.
If there is a need for you to have very tight memory controls, you could look into turning GC off entirely and managing memory yourself - See https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to-stop-worrying-and-love-the-heap-26c2462549a2/ for example
iirc jvm behaves similarly and doesn't return memory to the OS right away for perf reasons as Ian mentioned.


On Thursday, May 7, 2020 at 2:24:40 PM UTC-7, Ian Lance Taylor wrote:

On Thu, May 7, 2020 at 11:57 AM Naveen Kak <navee...@gmail.com> wrote:
>
> Ian,
> Any thoughts on this? Appreciate a response.

I'm sorry, I'm not sure what you want a response on.

Using top is a fine way to see total memory usage of your application.
It's a terrible way to examine the Go garbage collector.  Using
runtime.ReadMemStats will give you a better understanding of the
garbage collector.

Whether it makes sense to use sync.Pool depends on your application.
The sync.Pool documentation explains when it is useful.

Ian