memory limits with 6g-built executables

[kortschak]

Hi,

I'm starting to stress-test a genome aligner I've been porting from C+
+ to Go over the past few weeks (it's doing well and before
optimisation is only about 2x slower than the C++ - this will be made
up for by concurrent execution of parts and some other benefits from
the rewrite).

The initial stress test is to try performing alignment tasks on the
human genome (actually half of the human genome as we have the 2G
element limit on arrays/strings and the genome is ~3G). What I'm
finding is that during the data structure preparation the program dies
with an out of memory error:

runtime: out of memory: cannot allocate 1391591424-byte block
(15796600832 in use)
throw: out of memory

runtime.throw+0x40 /usr/local/src/go/src/pkg/runtime/runtime.c:102
runtime.throw(0x55c405, 0x52f19)
runtime.mallocgc+0x2f4 /usr/local/src/go/src/pkg/runtime/malloc.c:49
runtime.mallocgc(0x52f18430, 0x100000001, 0x7f6a00000001,
0x1e8000001e8, 0x441ea6, ...)
runtime.stringtoslicebyte+0x42 /usr/local/src/go/src/pkg/runtime/
string.c:255
runtime.stringtoslicebyte(0xfb9a9a2000, 0x7f6a52f18430,
0xf8497a1618, 0xf84002d800, 0x8000001e8, ...)
[...snip...]

I'm wondering where this limit is coming from - the C++ version
regularly allocates ~70-80GB (though not as single chunks) and the
system has available ~70GB:

$ free -g
total used free shared buffers
cached
Mem: 252 177 74 0
0 7
-/+ buffers/cache: 169 82
Swap: 1 0 1

Can anyone suggest either something that I'm overlooking or explain
limits that might exist?

thanks
Dan

[Russ Cox]

The memory allocator works on a contiguous block
of memory, which it reserves address space for when
the program starts. It reserves 16 GB and can't cope
with needing more than that.

You could change the limit by editing src/pkg/runtime/malloc.goc
Look for '16LL<<30'.

Russ

[John Arbash Meinel]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

What is the downside of increasing this? (Why not set it to 16e15 or 2^63?)

Is there a system limit to how much virtual space can be reserved?

Is there a reason the allocator has to use a contiguous block?

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk5BORYACgkQJdeBCYSNAAM6TQCgvXizp4dSQz8DA+UAZfP9NATr
XF8AnRa6hzwvo7lBinj2XADf9ydG+SuI
=UFRT
-----END PGP SIGNATURE-----

[Russ Cox]

> What is the downside of increasing this? (Why not set it to 16e15 or 2^63?)

It takes the address space away from other things,
like C code running in the same address space due to cgo.

> Is there a system limit to how much virtual space can be reserved?

Yes. 64-bit chips have 47-bit user address spaces.

> Is there a reason the allocator has to use a contiguous block?

It makes address calculations much easier.
In addition to the 16 GB of memory there is a
1 GB block below that is essentially a big bitmap
tracking properties of the 16 GB. Having it all
contiguous makes mapping from one to the other
very simple arithmetic.

Russ

[Dan Kortschak]

Thanks, I'll give that a go.

Is there any plan (or consideration) for a more dynamic approach to
reservation? - either compiler option/environment variable during build
or an equivalent to the environment variable GOMAXPROCS for setting this
limit during runtime (or other approaches).

thanks
Dan

[Russ Cox]

> Is there any plan (or consideration) for a more dynamic approach to
> reservation? - either compiler option/environment variable during build
> or an equivalent to the environment variable GOMAXPROCS for setting this
> limit during runtime (or other approaches).

I'd rather do the right thing automatically.
One possibility is to try to grow the space
every 16 GB. As long as nothing else has
taken it, we could then keep going.

Russ

[Russ Cox]

I created issue 2142.

[Dan Kortschak]

Yes, automatic is nicer and thanks for making that an issue.

Dan

[Dmitry Vyukov]

Why not start with, let's say, 64MB, and they grow as 128MB, 256MB...? 

[David Symonds]

On Wed, Aug 10, 2011 at 4:53 PM, Dmitry Vyukov <dvy...@google.com> wrote:

> Why not start with, let's say, 64MB, and they grow as 128MB, 256MB...?

It's only address space, not real memory, so there's no specific need
to conserve it. However, if we started out small and grew, then the OS
could allocate nearby pieces to other things (e.g. cgo-run code), and
so to increase our address space we'd need to get a non-contiguous
lump. That makes the address computations harder (we'd need lookup
tables, etc.), so we would want to minimise how much we need to do
that.

Dave.

[Dmitry Vyukov]

I deal with dynamic verification tools that reserve 16TB on a routine basis, and it does have problems with users/scripts/programs which use OS means to monitor memory usage. Since if(cgo) is already in the runtime, and exact policy to use does not make significant difference implementation-complexity-wise, I think gradual growing at least worth consideration.

[Jan Mercl]

On Wednesday, August 10, 2011 12:39:36 PM UTC+2, Dmitry Vyukov wrote:

I deal with dynamic verification tools that reserve 16TB on a routine basis

Do you really mean 2^44 B? Wow :-)

[Dmitry Vyukov]

Yup. Check out http://code.google.com/p/data-race-test/source/browse/trunk/gcc/rt/relite_defs.h

SHADOW_SIZE is 0x71C71C71C800ull bytes (114TB) that are mapped by mmap on startup.

[Dan Kortschak]

I've tried that now with a number of values (32, 33 and 34 - we can
cover those with our server) and I now don't get an out of memory error,
but I do get a segv unexpected address fault during a call to gob.Encode
(this was posted about a week ago with no responses, so I've tried a
number of changes and still get the same error). I can post the code
that causes the fault, but the data used are the first 750MB of the
human genome.

thanks
Dan

[⚛]

On Aug 9, 3:46 pm, Russ Cox <r...@golang.org> wrote:
> > Is there a reason the allocator has to use a contiguous block?
>
> It makes address calculations much easier.
> In addition to the 16 GB of memory there is a
> 1 GB block below that is essentially a big bitmap
> tracking properties of the 16 GB.  Having it all
> contiguous makes mapping from one to the other
> very simple arithmetic.

For small object sizes (up to, say, 1000 bytes), there exists the
following approach:

- The allocation chunk is 1 or 2**N pages (where N is a small number)

- The first (or last) 4/8 bytes of each chunk refer to additional
information about the chunk (bitmap, ...)

- Given an object pointer, finding the chunk's base-address is a
simple bit-mask operation

- If information about the object's size is absent, the type of the
chunk to which an arbitrary pointer belongs can be determined by a
page-table lookup. Because Go is typed and safe language, it is
probable that a high-performance garbage collector will know the
object size without the need for this kind of lookup.