confusion in heap management

[Michael Hudson-Doyle]

Hi all,

I have found a problem in how Go's heap is set up and managed.
Apologies for the length of this mail, it's a bit delicate :-)

It manifests on linux/arm64: when you statically compile a binary using
gccgo it fails to start up about 60% of the time with a message like
this:

runtime: address space conflict: map(0x7f72018000) = 0x7f69ef8000
fatal error: runtime: address space conflict

(the 60% failure rate turns out to be due to address space layout
randomization).

On investigation, this turns problem turns out to be in code that is
shared between the go distribution and gccgo, so it will affect any port
of the gc toolchain to linux/arm64, and further it seems to make sense
to talk about how to fix it here and get it fixed in the go toolchain
first.

The problem is this: Go has two approaches to managing the heap,
referred to in comments as "32-bit mode" and "64-bit mode". These names
are slightly misleading, and that's where the bug creeps in :-)

My understanding of how the modes operate:

In "32-bit mode", Go allocates 768 MB via mmap -- 256MB of this is a
bitmap, enough to track 2 GB, and 512MB is the initial heap. When more
heap is required, it calls mmap again and hopes that the newly allocated
memory is in the range that the bitmap can track. To make this more
likely, it tries to allocate the heap low in the process' address space,
but doesn't complain if it can't.

In "64-bit mode" the heap is mapped at an address of the form
0x0000XXc000000000 and is 128 Gb long. Because it conflicts with people
using ulimit -v, Go doesn't actually reserve this memory, just checks
that a small amount of memory at 0x0000XXc000000000 can be allocated,
and does a little dance to actually map the address space each time it
needs to use a new part of the heap (and hopes that nothing else has
intruded). If the attempt to allocate the heap fails, it falls back to
"32-bit mode".

So what's the problem? Well, it is that the check used for "64-bit
mode" is:

"sizeof(void*) and {pointer that lies within heap} >= 0XFFFFFFFFU"

but as well as being obscure, this can be _wrong_, when:

0) we are actually on a 64-bit system, and

1) the attempt to allocate the heap at 0x0000XXc000000000 fails, and

2) the attempt to allocate the heap at a low address fails, and the
kernel decides to allocate the heap somewhere up above
0x00000000ffffffff.

Then Go thinks it's in "64-bit mode" but really isn't and the "little
dance" I referred to that tries to reserve presumed-unreserved memory
doesn't work and the process bombs out with the "address space conflict"
message.

On linux/arm64, mapping 0x0000XXc000000000 is always going to fail
because addresses of this form are not valid:
https://github.com/torvalds/linux/blob/master/Documentation/arm64/memory.txt
and the attempt to allocate the heap at low addresse fails or succeeds
depening on ASLR stuff, so this is why I see the problem[1].

So if that's the problem, how to fix it? I proposed a CL for gccgo that
tracks explicitly whether the heap is reserved or not:
https://codereview.appspot.com/40610044/ which seems like an improvement
to me (I'm fairly sure the same patch would apply to the go distribution
with some path changes).

In addition, it would clearly be better to use "64-bit mode" for arm64.
Are the reasons for using an address with c0 00 in it still valid? (I
thought the GC was precise apart from the stack now, and I hope people
are not in the habit of storing lots of utf-8 on the stack). I guess
it's still sensible to pick a fixed address to reduce the chances of
other allocations getting in the way (or just mmap the full 128GB -- I
think people using ulimit -v will just fall back to "32-bit mode" here?
At least on Linux).

Cheers,
mwh

[Michael Hudson-Doyle]

Michael Hudson-Doyle <michael...@linaro.org> writes:

> On linux/arm64, mapping 0x0000XXc000000000 is always going to fail
> because addresses of this form are not valid:
> https://github.com/torvalds/linux/blob/master/Documentation/arm64/memory.txt
> and the attempt to allocate the heap at low addresse fails or succeeds
> depening on ASLR stuff, so this is why I see the problem[1].

Forgot my footnote:

[1] The reason why this shows up with static linking and not dynamic
linking on arm64 seems to be even more obscure, something to do with
the heuristics used to determine where in the address space to try
to allocate the heap, probably to do with where the stack ends up.

[Michael Hudson-Doyle]

Ping, any thoughts about this? Dmitriy seems the most likely victim
based on hg log...

[Russ Cox]

It is fine to keep a separate variable to track which mode is in use, and it is fine to try a lower address on the arm64 systems.

Russ