Why memory is not garbaged collected here?

[Ignacio Grande]

Hello everyone,

I've been playing a bit with Go to check the garbage collector, and I found an unexpected behaviour (unexpected for me). I don't know if it is a bug or something that I don't really understand.

Everything has been tested on tip version.

Program is here: http://play.golang.org/p/QbfGh5xnhF

package main

import (
    "time"
    //    "runtime/debug"
)

func main() {
    f()
    //    debug.FreeOSMemory()
    //    debug.FreeOSMemory()
    time.Sleep(10 * time.Hour)
}

func f() {
    a := make([]int, 100000000)
    for i := range a {
        a[i] = i
    }
}

The program allocates 763 MB of RAM (on 64-bits architecture) and then sleeps for 10 hours. I've been monitoring the memory used and the GC work by setting GODEBUG=gotrace=1

With the FreeOSMemory() lines commented, the memory is never GC'd nor released back to the system. I've waited about 30 minutes.

With one single line of FreeOSMemory(), the memory is not freed immediately (as I expected) but it seems that in the 2-3 next automatic GC runs it is finally found and freed, and eventually, released back to the OS.

With two FreeOSMemory() lines, the memory is released immediately (I suppose that in the second call).

I understand that this scenario is not very real. If I do some other work instead of just waiting, eventually the GC is executed, there is more garbage than my slice to be purged and everything is released.

Is this expected? In particular, that the 763 MB are not GC'd automatically without FreeOSMemery() lines.

Thanks

[Nacho]

As I understand, memory is returned to the OS about 5 minutes after is has been marked as free by the GC. And the GC runs every two minutes top, if not triggered by an increase in memory use. So worst-case would be 7 minutes to be freed.

In this case, I think that the slice is not marked as freed, but in use, so it would never be returned to the OS.

On Sun, May 18, 2014 at 12:43 AM, Stephen Gutekanst <stephen....@gmail.com> wrote:

I remember reading the GC does not return memory to the host OS unless you explicitly invoke FreeOSMemory() (but I think it's not important because of swap etc?).

Stephen

[k...@golang.org]

The behavior with one or two FreeOSMemory calls is expected.  The array gets freed after 1 GC, although for technical reasons it takes another GC for that fact to make it to the gctrace info.

With no FreeOSMemory calls, there's definitely a problem.  The array should get freed eventually as it does when there are FreeOSMemory calls.  I've opened bug 8018.

[Dmitry Vyukov]

On Sun, May 18, 2014 at 8:42 AM, <k...@golang.org> wrote:
> The behavior with one or two FreeOSMemory calls is expected. The array gets
> freed after 1 GC, although for technical reasons it takes another GC for
> that fact to make it to the gctrace info.

I think it's actually a bug. Gctrace does not output amount of
released memory, but if you look at top or Program Manager, the memory
must be released instantly. Filed:
https://code.google.com/p/go/issues/detail?id=8019

> With no FreeOSMemory calls, there's definitely a problem. The array should
> get freed eventually as it does when there are FreeOSMemory calls. I've
> opened bug 8018.
>
>
> On Saturday, May 17, 2014 3:47:35 PM UTC-7, Ignacio Grande wrote:
>>
>> As I understand, memory is returned to the OS about 5 minutes after is has
>> been marked as free by the GC. And the GC runs every two minutes top, if not
>> triggered by an increase in memory use. So worst-case would be 7 minutes to
>> be freed.
>>
>> In this case, I think that the slice is not marked as freed, but in use,
>> so it would never be returned to the OS.
>>
>>
>>
>>
>> On Sun, May 18, 2014 at 12:43 AM, Stephen Gutekanst
>> <stephen....@gmail.com> wrote:
>>>
>>> I remember reading the GC does not return memory to the host OS unless
>>> you explicitly invoke FreeOSMemory() (but I think it's not important because
>>> of swap etc?).
>>>
>>> Stephen
>>>
>>

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dmitry Vyukov]

OK, gctrace also prints this info.
With one FreeOSMemory call:

gc1(1): 9+0+173+0 us, 0 -> 0 MB, 20 (21-1) objects, 0/0/0 sweeps, 0(0)
handoff, 0(0) steal, 0/0/0 yields
gc2(1): 1+0+88+0 us, 0 -> 763 MB, 101 (103-2) objects, 15/0/0 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
gc3(1): 2+0+101+0 us, 763 -> 763 MB, 85 (108-23) objects, 24/0/0
sweeps, 0(0) handoff, 0(0) steal, 0/0/0 yields
scvg-1: 0 MB released
scvg-1: inuse: 763, idle: 0, sys: 764, released: 0, consumed: 763 (MB)

with two:

gc1(1): 12+0+232+0 us, 0 -> 0 MB, 20 (21-1) objects, 0/0/0 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
gc2(1): 1+0+97+0 us, 0 -> 763 MB, 100 (102-2) objects, 15/0/0 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
gc3(1): 1+0+120+0 us, 763 -> 763 MB, 84 (107-23) objects, 24/0/0
sweeps, 0(0) handoff, 0(0) steal, 0/0/0 yields
scvg-1: 0 MB released
scvg-1: inuse: 763, idle: 0, sys: 764, released: 0, consumed: 763 (MB)
gc4(1): 0+35+68+0 us, 0 -> 0 MB, 83 (107-24) objects, 26/0/25 sweeps,
0(0) handoff, 0(0) steal, 0/0/0 yields
scvg-1: 762 MB released
scvg-1: inuse: 0, idle: 763, sys: 764, released: 763, consumed: 0 (MB)