Using debug.FreeOSMemory() in real world

[Aaron Quint]

Hey All

We're now running a user facing part of our infrastructure with golang. And its great! Thanks for everyones hard work making it such a pleasure to work with.

Without going into the full details of the service (we hope to open source some day) it relies on a cgo wrapper around imagemagick (https://github.com/quirkey/magick) to do large amounts of image processing. We're seeing some weird things happen with memory as the process's RSS continues to grow and eventually starts moving into swap. Running pprof shows that the heap is much much smaller than the RSS of the process (25MB vs 500MB). 

Things I'm aware of:

1) We're working with images in memory which are relatively large (some can be >10MB's in size) so the process growing in leaps with these large allocations makes sense

2) Obviously go isn't managing the cgo code's memory, but in naive testing the c side of things was able to return memory to the OS.

3) I've tried profiling and getting data (locally) with MemStats and (in a live system) with ppprof/heap.

When doing some simple testing locally I noticed running `debug.FreeOSMemory()` did end up releasing a lot of memory back to the OS. 

Now after all that detail, the QUESTIONS:

1) Is it safe to run debug.FreeOSMemory() in a production system? Or am I doing it wrong?

2) Are there other flags or hints I can set to make the "background" reclaiming of memory happen faster? 

3) Is there any way to trace or instrument when these background cleanups are happening?

Thanks for your help

--AQ

[Dmitry Vyukov]

It is safe. The cost is somewhat larger than a normal garbage collection.

> 2) Are there other flags or hints I can set to make the "background"
> reclaiming of memory happen faster?

No (not counting recompiling Go)

> 3) Is there any way to trace or instrument when these background cleanups
> are happening?

You can set GODEBUG=gctrace=1 (for Go1.2) or GOGCTRACE=1 (prior to
Go1.2) env var, then the program will print info about GCs and memory
reclamation

[Caleb Spare]

Side note (not go-specific): I wouldn't run swap on a production server in most cases. It's usually better for your process to die and be restarted than to become glacially slow.

[Tamás Gulácsi]

2013. december 2., hétfő 18:23:11 UTC+1 időpontban Caleb Spare a következőt írta:

Side note (not go-specific): I wouldn't run swap on a production server in most cases. It's usually better for your process to die and be restarted than to become glacially slow.

On Monday, December 2, 2013 7:50:29 AM UTC-8, Aaron Quint wrote:

Hey All

We're now running a user facing part of our infrastructure with golang. And its great! Thanks for everyones hard work making it such a pleasure to work with.

Without going into the full details of the service (we hope to open source some day) it relies on a cgo wrapper around imagemagick (https://github.com/quirkey/magick) to do large amounts of image processing. We're seeing some weird things happen with memory as the process's RSS continues to grow and eventually starts moving into swap. Running pprof shows that the heap is much much smaller than the RSS of the process (25MB vs 500MB). 

Why not just carve out your image managing part into a separately callable program, and call properly parametrized (even can inherit file desriptors from parent)? This way you can be totally sure that all memory is released back to the OS, and a crash won't take down the master process.

[Aaron Quint]

Thanks, this was helpful. Rebuilt with 1.2 today and memory usage seems much more predictable so far. Also using GODEBUG on one of the nodes.

--AQ