Google Groups

memory limits with 6g-built executables

kortschak Aug 8, 2011 11:08 PM
Posted in group: golang-nuts

I'm starting to stress-test a genome aligner I've been porting from C+
+ to Go over the past few weeks (it's doing well and before
optimisation is only about 2x slower than the C++ - this will be made
up for by concurrent execution of parts and some other benefits from
the rewrite).

The initial stress test is to try performing alignment tasks on the
human genome (actually half of the human genome as we have the 2G
element limit on arrays/strings and the genome is ~3G). What I'm
finding is that during the data structure preparation the program dies
with an out of memory error:

runtime: out of memory: cannot allocate 1391591424-byte block
(15796600832 in use)
throw: out of memory

runtime.throw+0x40 /usr/local/src/go/src/pkg/runtime/runtime.c:102
        runtime.throw(0x55c405, 0x52f19)
runtime.mallocgc+0x2f4 /usr/local/src/go/src/pkg/runtime/malloc.c:49
        runtime.mallocgc(0x52f18430, 0x100000001, 0x7f6a00000001,
0x1e8000001e8, 0x441ea6, ...)
runtime.stringtoslicebyte+0x42 /usr/local/src/go/src/pkg/runtime/
        runtime.stringtoslicebyte(0xfb9a9a2000, 0x7f6a52f18430,
0xf8497a1618, 0xf84002d800, 0x8000001e8, ...)

I'm wondering where this limit is coming from - the C++ version
regularly allocates ~70-80GB (though not as single chunks) and the
system has available ~70GB:

$ free -g
             total       used       free     shared    buffers
Mem:           252        177         74          0
0          7
-/+ buffers/cache:        169         82
Swap:            1          0          1

Can anyone suggest either something that I'm overlooking or explain
limits that might exist?