Go Mobile and iOS Memory Management

[fire...@squareup.com]

Hey all,

I'm investigating Go Mobile, and to start I'm looking at how the Go runtime and GC behave inside of an iOS app. Some of my findings so far (correct me if I'm wrong on any of these):

• Unlike when a Go process runs directly within the OS, the memory it reserves seems to be real memory and not virtual memory (none of it is considered reclaimable), thus counting against the total memory for the process. This means that if the Go GC reserves 300MB and the process is only allowed 400MB before the OS kills it, then the rest of the process only has 100MB until the Go scavenger frees up some memory.
• The GC scavenger is reticent to return memory to the OS even if it's not in use. It does happen sometimes, but seems unpredictable, and primarily when GBs of memory are freeable.
• The Go GC does keep some overhead available internally. If you deallocate, then reallocate the same amount within the runtime it doesn't raise the total resident memory of the process. (So, it works.)
• Forcing a GC flush using `debug.FreeOSMemory` doesn't seem to do much.
• Memory pressure within the process doesn't seem to change anything.

The first two items are of the biggest concern, as they mean that the Go library could greatly increase the risk of out-of-memory (OOM) crashes for an app. I assume this is related to the fact that the Go runtime is running within another process and not talking directly to the OS. In testing on an iPhone 6 Plus (1GB RAM), the threshold really was something around 400-450MB of resident memory before the OS killed the app. Some possibilities I can think of to remedy this:

• Once memory is freed within the GC (but still reserved), mark it as clean/reclaimable again. Virtual memory would stay the same, but resident memory would drop. Not sure if this is possible.
• Monitor for memory pressure (lots of info on iOS memory here) and respond by scavenging. UIApplicationDidReceiveMemoryWarningNotification is an asynchronous option, and mach_vm_pressure_monitor/vm_page_free_count/vm_page_free_target may be useful as well.
• Scavenge more aggressively on mobile apps in general. Not sure what the performance tradeoffs would end up being, but given that Go was generally designed to work directly with the OS, and on desktop/server hardware that allows swapping, this might be a reasonable tradeoff.

Are any of these options reasonable to ameliorate memory pressure caused by Go's memory reservation? Are there others I'm missing? Is this something that's on the roadmap to be investigated?

My test app is available here if it's helpful. It's useful for observing some of the interaction between the Go runtime and the native runtime with respect to memory.

Thanks!

[David Crawshaw]

Hi. You say the virtual memory Go reserves appears to be real memory,
how are you determining that?

When I experimented with the 32-bit port about six months back, I was
able to reserve nearly as much virtual memory as the device had
physical memory, which goes far beyond what iOS will actually give an
app. So I went with the assumption that mmap was working correctly on
iOS. If you have evidence otherwise I can look into it.

Thanks,

d.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Eric Firestone]

My experimental method using the test app I linked:

- Allocate and initialize 300MB within the Go runtime.
- Free the 300MB
- See how much can be allocated using the native runtime before the OS kills the app. This only got to about 100MB.

In an independent run where I just allocate native memory the OS kills the app at around 400MB, which would indicate that the 300MB of Go memory was counting against the total.

One thing to note here is that I'm both allocating and initializing the memory in Go. If I just allocate the memory (an empty array for example) then the memory does seem to stay clean. Once it's been initialized and then freed it seems to stay dirty though.

[David Crawshaw]

Ah I see, you're actually using the memory.

I agree there are probably some elements of the runtime tuned to
large-memory systems that should be changed for low-memory systems.

+austin who was discussing OS memory on an issue recently: has there
been any discussion of scavenging more aggressively when low on
memory?

The most concerning point you bring up is that FreeOSMemory "doesn't
seem to do much". If you're no longer using memory, calling it should
do plenty. You may want to add a call to runtime.ReadMemStats in your
experiment and compare the state of Go's memory use after you run
FreeOSMemory.

If everything is working properly, adding a call to
runtime.FreeOSMemory inside
UIApplicationDidReceiveMemoryWarningNotification should be enough to
alleviate these issues. If it doesn't I think why it doesn't is worth
investigating.

[Austin Clements]

Hi Eric. Are you building for arm (32-bit) or arm64? Sadly, because of golang.org/issue/9993, we currently don't release memory back to the OS on arm64.

(I have some ideas for how to fix this, and since we're getting more and more platforms that use physical pages larger than 8K, we may have to tackle this issue in 1.7.)

On Wed, Jan 27, 2016 at 4:53 PM, David Crawshaw <craw...@golang.org> wrote:

Ah I see, you're actually using the memory.

I agree there are probably some elements of the runtime tuned to
large-memory systems that should be changed for low-memory systems.

+austin who was discussing OS memory on an issue recently: has there
been any discussion of scavenging more aggressively when low on
memory?

There has not. I'm not sure that's the right fix anyway, though it may be a useful last resort pressure valve. IMO, the scavenger is currently far too conservative. "Free memory" is an illusion. All memory is being used for one thing or another. If it's not being used by an application, it's being used by OS caches. Particularly in mobile (at least in Android; I think iOS does similar things), memory freed by one application can be used to cache whole other applications. Furthermore, if done right, it's cheap to return memory to the OS and pretty cheap to get it back. That means the benefits of returning memory are high and the costs are low.

[Austin Clements]

I don't know if this is possible on iOS, but another thing to check would be to set the environment variable GODEBUG to gctrace=1 and look at the stderr of the process. If it thinks it's releasing memory, there will be a line that says "scvg-1: NNN MB released"

[Eric Firestone]

Ah! Issue 9993 explains a lot. I am indeed running on arm64 and x86_64 (simulator on OS X). I've seen it return memory to the OS in the simulator before, but I think I've only seen it do so successfully one time. Other times I've been unable to get the resident memory to drop.

I gave gctrace=1 a shot, and it does indeed seem to be trying to release memory ("scvg-1: 70MB released" for example), but consistent with my previous findings, this doesn't lower the resident memory of the process. This was in the simulator, which is running on x86_64. I can try on-device as well (arm64), but that's been even more stubborn about returning memory and I suspect it's the same.

If we really can't reclaim memory from the Go runtime that might be a blocker to using Go Mobile. Any timing info you can provide on when this might be addressed (or options for patches, etc) is helpful.

Thanks!

[Austin Clements]

On Wed, Jan 27, 2016 at 6:06 PM, Eric Firestone <fire...@squareup.com> wrote:

Ah! Issue 9993 explains a lot. I am indeed running on arm64 and x86_64 (simulator on OS X). I've seen it return memory to the OS in the simulator before, but I think I've only seen it do so successfully one time. Other times I've been unable to get the resident memory to drop.

I gave gctrace=1 a shot, and it does indeed seem to be trying to release memory ("scvg-1: 70MB released" for example), but consistent with my previous findings, this doesn't lower the resident memory of the process. This was in the simulator, which is running on x86_64. I can try on-device as well (arm64), but that's been even more stubborn about returning memory and I suspect it's the same.

If we really can't reclaim memory from the Go runtime that might be a blocker to using Go Mobile. Any timing info you can provide on when this might be addressed (or options for patches, etc) is helpful.

Mind adding that justification to issue 9993? I think it hasn't been a priority just because nobody's made it a priority.

[fire...@squareup.com]

Posted in 9993 now that GitHub is back up. Is it worth filing a separate issue to investigate improving reclamation behavior in a mobile environment, such as making the scavenger more aggressive?

[Austin Clements]

I think it should be more aggressive in *all* environments. :) I'd say post that as a comment in 9993, just because we already have too many issues. If we wind up closing 9993 without addressing that, then we can pull it out as a separate issue at that point.