Off heap memory would be nice: Any chance to have that?

[Oliver Plohmann]

Hello,

in Java since JDK7 or so you can have off heap memory (f.ex. see article). This way you can put large amounts of data on the heap not being controlled by the GC (that is not causing memory to run full). I thought it would be kind of nice to have this in Go as well for special purposes, which in the times of BigData might not be that special any more, like large in-memory caches (see Apache Cassandra, Joss Infinispan, Hazelcast), some little in-memory Hadoop, etc.

The problem with the Go GC is also at the moment that you have no way to tell when the heap is running full. When it does so, the runtime crashes and you are basically sold. So putting MB of data on the heap is too risky. But the idea of off heap memory is not to work around GC issues till the GC has become more mature. For BigData applications and certain data-intensive server-side applications such a feature would be useful. Any chance to see something like that in Go one day?

Regards, Oliver

[Konstantin Khomoutov]

On Thu, 27 Mar 2014 07:08:15 -0700 (PDT)
Oliver Plohmann <saxo...@gmail.com> wrote:

> in Java since JDK7 or so you can have off heap memory (f.ex. see

> article<http://www.javacodegeeks.com/2012/12/escaping-the-jvm-heap-for-memory-intensive-applications.html>).
[...]

> The problem with the Go GC is also at the moment that you have no way
> to tell when the heap is running full. When it does so, the runtime
> crashes and you are basically sold. So putting MB of data on the heap
> is too risky. But the idea of off heap memory is not to work around
> GC issues till the GC has become more mature. For BigData
> applications and certain data-intensive server-side applications such
> a feature would be useful. Any chance to see something like that in
> Go one day?

You already can do that using anonymouns memory-mappings using [1]
(cross-platform or [2], [3]; maybe others).

1. https://github.com/edsrzf/mmap-go
2. https://github.com/riobard/go-mmap
3. https://launchpad.net/gommap/

[voidlogic]

This, use mmap and your own allocator.

[Oliver Plohmann]

Am Donnerstag, 27. März 2014 15:33:57 UTC+1 schrieb Konstantin Khomoutov:

You already can do that using anonymouns memory-mappings using [1]
(cross-platform or [2], [3]; maybe others).

1. https://github.com/edsrzf/mmap-go
2. https://github.com/riobard/go-mmap
3. https://launchpad.net/gommap/

Amazing ;-). Thanks.

[Dave Cheney]

The problem with the Go GC is also at the moment that you have no way to tell when the heap is running full. When it does so, the runtime crashes and you are basically sold. So putting MB of data on the heap is too risky.

[citation needed] how is this any different, or any worse than the JVM ? The Go runtime effectively works as if it were executed with -Xmx128G. Irrespective of the language or runtime used, if you're going to load a large amount of data into your program, some planning should be done before attempting this.

 

But the idea of off heap memory is not to work around GC issues till the GC has become more mature. For BigData applications and certain data-intensive server-side applications such a feature would be useful. Any chance to see something like that in Go one day?

What the JVM folks are doing, and what people in this thread have suggested is use mmap(2) to request your own anon memory, then manage it yourself. The GC will ignore any pointer that refernence memory that is not inside its working set.

Dave 

[Hamish Ogilvy]

Like the others have mentioned this is already part of Go. Memory mapping is a great solution, the other thing to do is use cgo. 

For mmap, the downside is needing to plan out non-fixed length data structures, data co-location on read (minimise page overhead), sparseness on deletes, RW locking and a few other gotchas. We've done this successfully with 10's of GB currently with virtually zero GC pause. This also loads basically instantly, so you have access to massive structures at near mem speed immediately. 

If you don't want to write your own mmap allocator (which can be really painful) and you have memory to burn, try cgo. We've used the following code up to ~1 billion structs under heavy RW operation for months without any issues and very little GC pause. Note: We lock at a "Term" level, e.g. outside this code.

https://gist.github.com/mish15/9822474

[Dmitry Vyukov]

It's very easy to have effectively "off the heap" memory in Go:

buf := make([]byte, 16<<30)

This is 16GB of memory that you can split into objects as you wish
(using unsafe), and it will have virtually zero effect on GC.

[Dmitry Vyukov]

You can't store pointers back into heap in this array. But it's
reasonable, because otherwise it would have to be scanned by GC.