How to detect memory leaks?

[Markus Zimmermann]

Hi gophers!

I am doing a lot of automatic testing lately (to be exact, generation of automatic tests) and one category I am especially interested in is detecting memory leaks. I have three questions I need guidance with:

1. How can memory leaks be detected? If a function foo() is called, how can I deterministically detect that allocations/objects got leaked by this function or due to this function?
2. How can allocations/objects be marked? If the function foo() leaks an object of type bar, how can I state that such an object is leaking?
3. How can I trace a leaked object back to its location? Meaning, is it possible to get a stack trace of the allocation of a particular object?

I am wondering if the first question could be easily answered using runtime.GC() and runtime.ReadMemStats(m *MemStats). Please consider the following program http://play.golang.org/p/0poBP631fc When I execute the given program I get the following output.

$ go version
go version go1.4 linux/amd64
$ go run -gcflags=-m tt.go
# command-line-arguments
./tt.go:24: can inline func·001
./tt.go:24: func literal escapes to heap
./tt.go:22: moved to heap: ch
./tt.go:25: &ch escapes to heap
./tt.go:22: make(chan bool, 0) escapes to heap
./tt.go:22: make(chan bool, 0) escapes to heap
./tt.go:21: <S> func literal does not escape
./tt.go:18: <S> &stats[1] does not escape
./tt.go:20: <S> func literal does not escape
./tt.go:38: <S> &stats[2] does not escape
./tt.go:10: main make([]runtime.MemStats, 4) does not escape
./tt.go:13: main &stats[0] does not escape
./tt.go:15: main func literal does not escape
./tt.go:43: main &stats[3] does not escape
./tt.go:48: main ... argument does not escape
Inner leaks 0, outer leaks 1                                                                                 
Inner leaks 0, outer leaks 0                                                                                 
Inner leaks 0, outer leaks 0                                                                                 
Inner leaks 0, outer leaks 0

I have three questions regarding this program:
1. Does it make sense to use the runtime functions in this way?
2. Why is there a “leak” in the first iteration and how do I debug it?
3. Why does the line “./tt.go:22: make(chan bool, 0) escapes to heap” show up twice?

Cheers,
Markus

[Ian Lance Taylor]

On Wed, Dec 31, 2014 at 9:05 AM, Markus Zimmermann <zim...@gmail.com> wrote:
>
> I am doing a lot of automatic testing lately (to be exact, generation of
> automatic tests) and one category I am especially interested in is detecting
> memory leaks. I have three questions I need guidance with:
>
> 1. How can memory leaks be detected? If a function foo() is called, how can
> I deterministically detect that allocations/objects got leaked by this
> function or due to this function?

I'm not sure what you mean by leaks. Do you mean allocations (memory
that will have to be cleaned up by the garbage collector) or do you
mean permanent leaks (memory that is retained in some global variable
and will therefore never be collected).

For allocations, see http://golang.org/pkg/testing/#AllocsPerRun .

> 3. How can I trace a leaked object back to its location? Meaning, is it
> possible to get a stack trace of the allocation of a particular object?

Specifically, no, but for typical general uses, see
http://golang.org/pkg/runtime/pprof/ and
http://blog.golang.org/profiling-go-programs .

The Mallocs field measures the number of total objects. The Frees
fields measures the number of objects that have been allocated in
memory but are not live. Depending on what you want to measure it may
suffice to just measure Mallocs and not worry about Frees.

> 2. Why is there a “leak” in the first iteration and how do I debug it?

I would guess it's the allocation of the channel.

> 3. Why does the line “./tt.go:22: make(chan bool, 0) escapes to heap” show
> up twice?

I don't know.

Ian

[Dave Cheney]

On point three, I think the answer is you get two lines of output because that anon function is compiled twice, once for the function itself, and once for the inlined form of the function.

[Markus Zimmermann]

On 31.12.2014 09:33:02 you wrote:
> On Wed, Dec 31, 2014 at 9:05 AM, Markus Zimmermann <zim...@gmail.com> wrote:
> > I am doing a lot of automatic testing lately (to be exact, generation of
> > automatic tests) and one category I am especially interested in is
> > detecting memory leaks. I have three questions I need guidance with:
> >
> > 1. How can memory leaks be detected? If a function foo() is called, how
> > can
> > I deterministically detect that allocations/objects got leaked by this
> > function or due to this function?
>
> I'm not sure what you mean by leaks.  Do you mean allocations (memory
> that will have to be cleaned up by the garbage collector) or do you
> mean permanent leaks (memory that is retained in some global variable
> and will therefore never be collected).

I mean permanent leaks which will not be collected by the GC due to a bug in the code. For example consider the implementation of the "remove" method of the container/list package http://golang.org/src/container/list/list.go#L108 We can introduce a leak here by not removing the pointers to the element e.g. removing line 110-114. If we then do a black box test that adds and removes one element and then tests if Len() returns 0 and Front() and Back() return nil, we will get a passing test. However, the list structure is corrupted and the element will not be collected. A "check for leaks" call around the add and remove part of the test would find the problem http://play.golang.org/p/GNhehblFB2 Sure, better tests or white box testing (like list_test.go does with "checkListPointers") would find this bug but it is not always that easy. Tools to isolate such leaks (e.g. a remove method that does not remove all instances of an object, a clean method that does not clean everything up) faster would be in my opinion helpful.

> For allocations, see http://golang.org/pkg/testing/#AllocsPerRun .
>
> > 3. How can I trace a leaked object back to its location? Meaning, is it
> > possible to get a stack trace of the allocation of a particular object?
>
> Specifically, no, but for typical general uses, see
> http://golang.org/pkg/runtime/pprof/ and
> http://blog.golang.org/profiling-go-programs .

I am wondering if the mechanisms of GODEBUG="allocfreetrace=1" or something similar could be used via code. A function could be called when an object is created. The function could then remember which object was created. When it is time to check for leaks a heap dump could be searched for the remembered object. If it is still there, it must be a leak. Do you think that this would be a viable strategy?

I want to measure how many objects have been allocated between two marks which are still alive after an explicit GC run.

> > 2. Why is there a “leak” in the first iteration and how do I debug it?
>
> I would guess it's the allocation of the channel.

Since the channel is not in scope anymore, shouldn't it be then collected by a call to runtime.GC()? And shouldn't Mallocs and Frees then cancel each other out? Also the whole code is executed four times but this allocation/free difference is only in the first iteration. If I do something similar without a goroutine http://play.golang.org/p/F3BP4pc-uu I do not get any allocation/free differences. Does this maybe mean that the allocation is done because of the goroutine?

PS: Happy new year!

[Markus Zimmermann]

On Thursday, January 1, 2015 12:15:06 AM UTC+1, Dave Cheney wrote:

On point three, I think the answer is you get two lines of output because that anon function is compiled twice, once for the function itself, and once for the inlined form of the function.

Why compile the function if it is already inlined? Is this a bug or do I not understand the concept?

[minux]

The compiler still need to compile the function body that is inlined, and the make

in the new function still escapes, hence the second message. 

[Dave Cheney]

Because it may be called in a context where it cannot be inlined. 

[Ian Lance Taylor]

What you've described is not what I would consider to be a permanent
leak that will not be collected by the GC. Failing to clear those
fields means that the list may be held as long as the caller holds the
element passed to the Remove method. It's not good, because it holds
unnecessary memory, but it's not a permanent leak.

To test that kind of leak, I think I would write a benchmark and look
at the MemBytes field in BenchmarkResult.

>> For allocations, see http://golang.org/pkg/testing/#AllocsPerRun .
>>
>> > 3. How can I trace a leaked object back to its location? Meaning, is it
>> > possible to get a stack trace of the allocation of a particular object?
>>
>> Specifically, no, but for typical general uses, see
>> http://golang.org/pkg/runtime/pprof/ and
>> http://blog.golang.org/profiling-go-programs .
>
> I am wondering if the mechanisms of GODEBUG="allocfreetrace=1" or something
> similar could be used via code. A function could be called when an object is
> created. The function could then remember which object was created. When it
> is time to check for leaks a heap dump could be searched for the remembered
> object. If it is still there, it must be a leak. Do you think that this
> would be a viable strategy?

Assuming I'm understanding you correctly, it's normal for memory to
leak in that sense. A value referenced by a global variable has not
leaked. So it does not make sense to use this approach
indiscriminately.

In general I don't see this as a useful technique to apply as part of
routine testing. Routine testing should show a potential memory
allocation problem. Then techniques like setting GODEBUG can be used
to debug the problem. I suspect that automatically dumping out a
trace for every allocated memory block will produce an unhelpful level
of detail when not combined with something like the heap profiler.

> I want to measure how many objects have been allocated between two marks
> which are still alive after an explicit GC run.

Use the MemAllocs field of BenchmarkResult.

>> > 2. Why is there a “leak” in the first iteration and how do I debug it?
>>
>> I would guess it's the allocation of the channel.
>
> Since the channel is not in scope anymore, shouldn't it be then collected by
> a call to runtime.GC()? And shouldn't Mallocs and Frees then cancel each
> other out? Also the whole code is executed four times but this
> allocation/free difference is only in the first iteration. If I do something
> similar without a goroutine http://play.golang.org/p/F3BP4pc-uu I do not get
> any allocation/free differences. Does this maybe mean that the allocation is
> done because of the goroutine?

Fair points, it could well be something allocated by the scheduler
somewhere.

Ian