Go doesn't return memory to system

[Alex]

Hi Gophers,

I have a problem with memory consumption. Looks like Go doesn't free memory which isn't relevant anymore. You can find example code on golang play (link below) this is just sketch which doesn't do anything but yet it consumes a lot of memory. Idea of this algo is to keep in memory only data of current iteration and previous one. Before next iteration just use data of current iteration as "previous".

Any ideas?

http://play.golang.org/p/ocoiXR4Oxz

Below is output for real problem size.

> go run tsp.go

Setup is done

Aprev size 276 152 Mb

Aprev size 2024 152 Mb

Aprev size 10626 152 Mb

Aprev size 42504 152 Mb

Aprev size 134596 152 Mb

Aprev size 346104 177 Mb

Aprev size 735471 314 Mb

Aprev size 1307504 466 Mb

Aprev size 1961256 720 Mb

Aprev size 2496144 1197 Mb

Aprev size 2704156 1385 Mb

Aprev size 2496144 1542 Mb

Aprev size 1961256 1579 Mb

Aprev size 1307504 1579 Mb

Aprev size 735471 1579 Mb

Aprev size 346104 1579 Mb

Aprev size 134596 1579 Mb

Aprev size 42504 1579 Mb

Aprev size 10626 1579 Mb

Aprev size 2024 1579 Mb

Aprev size 276 1579 Mb

Aprev size 24 1579 Mb

Aprev size 1 1579 Mb

Min is 74 time 14.074597s sec

[Alex]

Basically my question is how can I force Go to clear memory allocated for a map which I don't need any more?

Alex

[Martin Angers]

This is because the memory freed by the GC is not returned to the system, it is reused by Go if needed. Your memory IS freed by Go, you just don't see it because you use the memstats.Sys field. Here is an example, side-by-side, which shows Sys and Alloc (actual memory in use by the program):

http://play.golang.org/p/o5U7GOezhd

Note that I forced a GC before printing the stats, because the example was too short otherwise (no collect was needed yet).

[Alex]

OK. But will it return this memory to the system? I mean after some time, not only after program stop. Or maybe there is some method to let GC know that it can free unused memory?

[Martin Angers]

I would guess that this is a configurable limit, but I don't know enough about the GC's internals to answer that question. I doubt there is a Go stdlib method to do this, since this is the kind of thing you shouldn't have to worry about. If you're interested in diving into the matter, you can browse the GC's code here:

http://code.google.com/p/go/source/browse/src/pkg/runtime/

Mostly mgc0.c/.h, I think, and possibly the other mem* files.

[Kevin Gillette]

I seem to recall either that tip now releases pages back to the kernel, or that there's discussion of making that change.

[Martin Angers]

Too bad the playground doesn't seem to work on tip.golang.org! It could be useful to quickly test some things like that.

[John Graham-Cumming]

On Monday, January 21, 2013 3:19:27 PM UTC, Alex wrote:

OK. But will it return this memory to the system? I mean after some time, not only after program stop. Or maybe there is some method to let GC know that it can free unused memory?

 In mheap.c (in pkg/runtime) there's a function called MHeap_Scavenger that contains the following comment:

   // If a span goes unused for 5 minutes after a garbage collection,                

   // we hand it back to the operating system. 

Spans are released by calling runtime.SysUnused which will call madvise with MADV_DONTNEED.

John.

[Alex]

The same situation on tip. Just tried.

[Rémy Oudompheng]

On 2013/1/21 Alex <alex....@gmail.com> wrote:
> The same situation on tip. Just tried.
>

Your program artificially abuses GC's heuristics for triggering times.
Try running with a lower value of the environment variable GOGC, for
example GOGC=40.

Rémy.

[Dave Cheney]

Please run your program with GOGCTRACE=1 set and post the results.
That will show the scavenger in operation and if it is able to return
memory to the operating system.

Two things to note.

1. Go uses a non compacting GC. Pages cannot be returned to the
operating system unless they are completely unused. Memory
fragmentation is large part of the scavengers ability to operate.

2. The scavenger uses various strengths of madvise(2) on various Unix
platforms. As the name would suggest, this is advice, not an edict,
the operating system can/will ignore this advice. The
runtime.SysUnused function is not implemented on some platforms, so on
those platforms the scavenger will have no effect.

> --
>
>

[Alex]

GOGC=30 doesn't help

> export GOGC=30

> go run tsp.go 

Setup is done

Size 2 276 instances 127 Mb 99 Mb

Size 3 2024 instances 127 Mb 99 Mb

Size 4 10626 instances 127 Mb 101 Mb

Size 5 42504 instances 127 Mb 82 Mb

Size 6 134596 instances 127 Mb 101 Mb

Size 7 346104 instances 176 Mb 152 Mb

Size 8 735471 instances 287 Mb 251 Mb

Size 9 1307504 instances 467 Mb 417 Mb

Size 10 1961256 instances 717 Mb 638 Mb

Size 11 2496144 instances 951 Mb 854 Mb

Size 12 2704156 instances 1133 Mb 980 Mb

Size 13 2496144 instances 1306 Mb 991 Mb

Size 14 1961256 instances 1306 Mb 878 Mb

Size 15 1307504 instances 1306 Mb 677 Mb

Size 16 735471 instances 1306 Mb 469 Mb

Size 17 346104 instances 1306 Mb 523 Mb

Size 18 134596 instances 1306 Mb 543 Mb

Size 19 42504 instances 1306 Mb 549 Mb

Size 20 10626 instances 1306 Mb 551 Mb

Size 21 2024 instances 1306 Mb 551 Mb

Size 22 276 instances 1306 Mb 551 Mb

Size 23 24 instances 1306 Mb 551 Mb

Size 24 1 instances 1306 Mb 551 Mb

Min is 26442 time 5m18.048446s sec

Memory consumed from the system still doesn't return back to the system. And BTW Allocated memory also behaves quite strange. On previous attempt it allocated much less memory. And as I can judge on instance count memory footprint should drop to 99Mb

[Alex]

I removed runtime.GC() - seems that is the cause of not decreasing Allocated memory.

[Alex]

This are two traces made with GOGCTRACE=1 and GOGC=30. First one without runtime.GC() after each outer loop iteration. And second one - with runtime.GC() after each outer loop iteration.

https://gist.github.com/de4a4c6e409c7e7b539c

[Felix Sun]

So I get the same problem, Anyway to let the Go process return the memory to system?

[Dmitry Vyukov]

runtime/debug.FreeOSMemory()

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[John Nagle]

On 8/16/2013 7:18 AM, Dmitry Vyukov wrote:
> runtime/debug.FreeOSMemory()

Without a compacting garbage collector, that may not recover
much space. But give it a try. If you do mostly big allocations,
you might get space back.

John Nagle