Q about memory allocation/cleanup with interfaces

[Christopher Crotty]

Apologies in advance if this is not the right group to post this...

I am having a memory leak issue in an app that reads data reports from a sensor and stores them for a length of time.  Running pprof on the app, It appears that the memory is not getting released when the old data is purged.  

There can be several report types, so I created an interface definition and am storing these reports under a slice of interfaces in a map with a timestamp key

When it comes time to purge, I'm setting the slice at that timestamp to nil, but the memory does not seem to be reclaimed.

Dirty Details:

The data comes in as a string message which I unmarshal into one of the concrete report types.  The reports are stored in the following structure...

map[string]map[int64][]ReportIface  where the first map has a key of sensorID and the inner map has a key of timestamp.  The ReportIface is an interface definition that all of the reports implement

During purging, I roll through the inner map and if the timestamp key is less than the purge time I  run the following

if reports[timestamp] != nil {

  reports[timestamp] = nil

}

delete(reports, timestamp).

Is there anything else I need to do for this?   Pprof is showing the memory from the unmarshal as still in use even after the purge.

Thanks in advance,

Chris

[Jesper Louis Andersen]

It is perhaps easiest if I go through the conditions which must be met for data to be reclaimed by the Operating System:

First, the data must be unreachable by goroutines through a path of pointers. When you nil the object, you break such a path, but you must make sure there are no other such paths to the object. Once there is no reachability of the object, it is eligible for collection.

Second, garbage collection must run. It runs periodically, but it can take some time before it does so, and if you check memory usage right after you've freed up data, it might not have run yet. You can run Go with GC debug output (env GODEBUG=gctrace=1 ...) to see when it actually runs. This is a property of using a mark'n'sweep garbage collection strategy as the memory is scavenged for unreachable data immediately. It is somewhat in contrast to reference counting methods which will usually free up the memory quicker.

Third, the GC must have some excess data it wants to give back to the OS. Usually it likes to reuse a memory space if it is needed shortly and will only give memory back eventually if the program consistently runs in less memory.

There are some subtle things you might want to look out for however. When you unmarshal from the string into the concrete report type, you should probably avoid keeping a slice to the underlying source string. Otherwise, it will keep said string alive for the duration of the reports existence. Explicitly copying data and making sure you have no reference to the source string is one thing that is important to get right, especially if the source string is much larger than the report in memory size. Another thing is that I would just do `delete(reports, timestamp)` since it is a no-op if timestamp doesn't exist, and deletion effectively breaks the pointer anyway.

I hope this helps somewhat in narrowing down what could be wrong.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d0a1eee7-7968-4560-9cd4-98d69e48ea91n%40googlegroups.com.

--

J.

[Jesper Louis Andersen]

On Sun, Sep 13, 2020 at 1:42 PM Jesper Louis Andersen <jesper.lou...@gmail.com> wrote:

Second, garbage collection must run. It runs periodically, but it can take some time before it does so, and if you check memory usage right after you've freed up data, it might not have run yet. You can run Go with GC debug output (env GODEBUG=gctrace=1 ...) to see when it actually runs. This is a property of using a mark'n'sweep garbage collection strategy as the memory is scavenged for unreachable data immediately. It is somewhat in contrast to reference counting methods which will usually free up the memory quicker.

Too fast. Memory is NOT scavenged immediately in a mark'n'sweep collector.