mmapping over Go heap

[Nick Zavaritsky]

Hi,

Packages offering memory mapped files in golang follow one of the two possible routes:

1) memory is accessed via io.Reader (e.g. golang.org/x/exp/mmap);

2) mapped memory is directly exposed as []byte; accessing it after munmap leads to SIGSEGV or worse (e.g. github.com/edsrzf/mmap-go).

I'm not happy with option #1 due to the added overhead. There are multiple high performance interfaces in Linux based on memory mappings such as io-uring, ebpf ring buffers, and AF_XDP.. I'd love to avoid extra copies, call overhead and *especially* heap allocations.

On the other hand, option #2 makes me concerned about dangling pointers that remain after the memory is unmapped.

I'm curious if the following approach might work.

1) obtain []byte from golang heap;

2) mmap on top.

Munmap will turn the address range into anonymous memory mapping again, similar to other pages backing golang heap. Dangling pointers, if any, are safe to follow. Once GC proves that no references remain, the address range naturally becomes available for reuse.

I'm not familiar with the runtime internals. Please let me know if this is a bad idea.

Best,

N

[Brian Candler]

> accessing it after munmap leads to SIGSEGV or worse

There must be a hundred different ways you can make a Go program segfault, and this doesn't sound any worse that the rest of them.

Besides, unless the GC is changed to handle mmap regions differently, I think you would have the problem the opposite way around. What if you have an area of heap memory which has been mmap'd, and you don't call munmap, but there are no pointers left referring to it? Wouldn't the heap quietly start allocating other objects from that address space?