Unable to allocate (large) memory

[Glen Newton]

Hello,

I run this program on one of our new huge servers (1 TB ram, 40 core (10core x 4 cpu) Xeon E5-4650 v2, CentOS release 6.5, go version go1.4 linux/amd64 ):

https://play.golang.org/p/yAJPIu-WSf

$ /usr/bin/time -f "%M" ./goAllocTest 

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

runtime: out of memory: cannot allocate 8589934592-byte block (128850067456 in use)

fatal error: out of memory

runtime stack:

runtime.throw(0x545e2a)

        /usr/local/go/src/runtime/panic.go:491 +0xad

runtime.largeAlloc_m()

        /usr/local/go/src/runtime/malloc.c:390 +0xee

runtime.onM(0x54bab0)

        /usr/local/go/src/runtime/asm_amd64.s:257 +0x68

runtime.mstart()

        /usr/local/go/src/runtime/proc.c:818

goroutine 1 [running]:

runtime.switchtoM()

        /usr/local/go/src/runtime/asm_amd64.s:198 fp=0xe0080e0db0 sp=0xe0080e0da8

runtime.mallocgc(0x1ffffffd0, 0x49e020, 0x1, 0x54be40)

        /usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xe0080e0e60 sp=0xe0080e0db0

runtime.newarray(0x49e020, 0x3ffffffa, 0x7f9d7adb0000)

        /usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xe0080e0e98 sp=0xe0080e0e60

runtime.makeslice(0x499ee0, 0x3ffffffa, 0x3ffffffa, 0x0, 0x0, 0x0)

        /usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xe0080e0ee0 sp=0xe0080e0e98

main.main()

        /home/newtong/work/lavidallocate/go/goAllocTest/main.go:19 +0x19b fp=0xe0080e0f98 sp=0xe0080e0ee0

runtime.main()

        /usr/local/go/src/runtime/proc.go:63 +0xf3 fp=0xe0080e0fe0 sp=0xe0080e0f98

runtime.goexit()

        /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xe0080e0fe8 sp=0xe0080e0fe0

36239904

'top' indicates that this process had 127GB virtual, 8.6GB resident just before it failed. The above /usr/bin/time indicates 36GB maximum resident.

 

I don't understand why this failed to early: I would have expected it to continue allocating a lot longer. 'top' at count=13 indicated:

Mem:  1058921992k total, 20655084k used, 1038266908k free,   310636k buffers

15705 newtong  20   0  127g 8.6g  644 S 106.5  0.9   0:15.81 goAllocTest                                                                 

So it looks to me that there is a lot of memory available.

Is there a problem with the above code? My calculations is that the initial alloc of for the 'var a []*[]int' is 2147483637*64b (64 bit pointer, correct?) = 2147483637*8B = 16GB; then each subsequent alloc for the '[]int' is 2147483637*32b = 2147483637*4B=8GB.

So initial alloc of 'a' plus 14 allocs of []int = 32GB + 14*8GB = 144GB

When I run this on my laptop with only 16GB ram (Fedora18), I get the same issue _at_the_same_point (14 allocs). This suggests that either my code is wrong and I am not understanding things or....?

'top' just before failing on laptop:

KiB Mem:  16373032 total,  5681960 used, 10691072 free,   508400 buffers

 5998 newtong   20   0  119g 1.1g  644 S 105.1  7.2   0:20.82 goAllocTest                                                                 

Both fail close to 127GB virtual: is this a magic number wrt the memory model?

Any help in getting me to understand this would be appreciated.

Thanks,

Glen

[Chris Kastorff]

First, yes, you are hitting the Go VM limit for 64-bit programs, which
is a bit underneath 128GB per process. A Go developer may be able to
explain why that limit was chosen and if it's tunable; I don't know.

Second, to explain why it works the same without physical ram to back it:
You're allocating space, but you're never using any of it. The kernel's
virtual memory manager never gets a request to write to almost all of
the pages you're creating, so it doesn't bother mapping a physical page
to the virtual address space in the program, so you don't run out of
physical pages.

> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com
> <mailto:golang-nuts...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[gol...@icarus.freeuk.com]

There is a blog post http://utcc.utoronto.ca/~cks/space/blog/programming/GoBigVirtualSize on the subject.

On Tuesday, December 23, 2014 8:11:36 AM UTC-8, Glen Newton wrote:

Hello,

I run this program on one of our new huge servers (1 TB ram, 40 core (10core x 4 cpu) Xeon E5-4650 v2, CentOS release 6.5, go version go1.4 linux/amd64 ):

https://play.golang.org/p/yAJPIu-WSf

[snip]                                     

[Daniel Theophanes]

I would recommend you read the following from the go source file:

https://github.com/golang/go/blob/release-branch.go1.4/src/runtime/malloc.h#L152

Adjusting the constants above the linked line should change your max limits. I *think* you can safely adjust it to other sane values.

-Daniel

On Tuesday, December 23, 2014 8:11:36 AM UTC-8, Glen Newton wrote:

[Glen Newton]

Thank-you all for your responses: I understand this behaviour now.

So a follow-up: we have applications (C++, Java, etc) that need several hundred GB to operate (big bioinformatics apps). If I want to implement these types of apps in Go, and the underlying algorithm is similar (needing this kind of space), how should I proceed if I am going to need, say 1/2 TB of ram? 

It seems that I a locked out of writing Go programs that need this kind of memory. I would prefer this was not the case. :-(

Is there a time frame for when this limit will not be the default in Go?

[I would prefer not to hack the Go compiler with unintended consequences...]

---

Below is a similar similar Java application that also allocates in a similar fashion. I wrote just as a reality check.

It creates 121 int[] before OOM. Just before it fails, 'top' says it is 1TB resident:

Tasks: 1401 total,   1 running, 1400 sleeping,   0 stopped,   0 zombie

Cpu(s): 58.2%us,  1.3%sy,  0.0%ni, 40.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  1058921992k total, 1056937768k used,  1984224k free,    18256k buffers

Swap:  1023992k total,   165292k used,   858700k free,    85812k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

21468 newtong  20   0     ? 1.0t 8080 S 4771.9 99.0  15:34.55 java  

public class AllocTest{

    public static final void main(String[] args){

        AllocTest at = new AllocTest();

        at.start();

    }

    public void start(){

        int max = Integer.MAX_VALUE-10;

        System.out.println(max);

        int[][] z = new int[max][];

   

        // Testing showed OOM at sub~=121                                                                                              

        int sub = 130;

        for(int i=0; i<sub; i++){

            System.out.println(max + " " + i);

            try{

                z[i] = new int[max];

            }catch(Throwable t){

                t.printStackTrace();

                return;

            }

        }

    }

}

Thanks,

Glen

[Michael Jones]

Setting a constant and recompiling is neither hard nor hacking. Be bold.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Daniel Theophanes]

Hi Glen,

Go can handle it.

Pull the git source and update to 1.4

git clone https://github.com/golang/go

git checkout go1.4

git clean -f

In file: src/runtime/malloc.h

Change line 135 to:

MHeapMap_Bits = 40 - PageShift,

cd src

./all.bash

-Daniel

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/Ij1uf9ql8Zk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

[Glen Newton]

Ha-ha!  :-)

I had assumed that it was more than a constant and various complicated things happen in various places using this value (did not check the code beyond that particular code point). But if it _is_ just a simple constant, then I will _be_bold_ and will do as you suggest over the next couple of weeks and report back how things have gone!  :-)

Thanks,

Glen   

[Glen Newton]

I followed your instructions, and recompiled the application. On my laptop, getting:

$ ./all.bash

# Building C bootstrap tool.

.

. <stuff removed>...

.

Installed Go for linux/amd64 in /home/newtong/work/go

Installed commands in /home/newtong/work/go/bin

*** You need to add /home/newtong/work/go/bin to your PATH.

$ export PATH=/home/newtong/work/go/bin:$PATH

$ which go

~/work/go/bin/go

$ cd ~/work/goAllocTest

$ rm goAllocTest

$ go build

$ /usr/bin/time -f "%M" ./goAllocTest 

0

1

2

3

4

5

6

runtime: out of memory: cannot allocate 17179869184-byte block (120260132864 in use)

fatal error: out of memory

runtime stack:

runtime.throw(0x545e2a)

/usr/local/go/src/runtime/panic.go:491 +0xad

runtime.largeAlloc_m()

/usr/local/go/src/runtime/malloc.c:390 +0xee

runtime.onM(0x54bab0)

/usr/local/go/src/runtime/asm_amd64.s:257 +0x68

runtime.mstart()

/usr/local/go/src/runtime/proc.c:818

goroutine 1 [running]:

runtime.switchtoM()

/usr/local/go/src/runtime/asm_amd64.s:198 fp=0xde0809cdb0 sp=0xde0809cda8

runtime.mallocgc(0x3ffffffa8, 0x49e020, 0x1, 0x54be40)

/usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xde0809ce60 sp=0xde0809cdb0

runtime.newarray(0x49e020, 0x7ffffff5, 0x7f4e81da3000)

/usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xde0809ce98 sp=0xde0809ce60

runtime.makeslice(0x499ee0, 0x7ffffff5, 0x7ffffff5, 0x0, 0x0, 0x0)

/usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xde0809cee0 sp=0xde0809ce98

main.main()

/home/newtong/work/lavidallocate/go/goAllocTest/main.go:19 +0x19b fp=0xde0809cf98 sp=0xde0809cee0

runtime.main()

/usr/local/go/src/runtime/proc.go:63 +0xf3 fp=0xde0809cfe0 sp=0xde0809cf98

runtime.goexit()

/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xde0809cfe8 sp=0xde0809cfe0

1174816

Suggestions?

Thanks,

Glen

[Dave Cheney]

The stack trace shows remnants of your old Go install in /usr/local/go. Did you have GOROOT set ?

[Glen Newton]

Doh!

Yes, that was it.

Now on the 1TB server I am getting:

$ /usr/bin/time -f "%M" ./goAllocTest

0

1

2

.

.<stuff removed>...

.

60

61

62

runtime: out of memory: cannot allocate 17179869184-byte block (1082332807168 in use)

fatal error: out of memory

runtime stack:

runtime.throw(0x545e2a)

        /home/newtong/work/go/src/runtime/panic.go:491 +0xad

runtime.largeAlloc_m()

        /home/newtong/work/go/src/runtime/malloc.c:390 +0xee

runtime.onM(0xd040014400)

        /home/newtong/work/go/src/runtime/asm_amd64.s:257 +0x68

runtime.mstart()

        /home/newtong/work/go/src/runtime/proc.c:818

goroutine 1 [running]:

runtime.switchtoM()

        /home/newtong/work/go/src/runtime/asm_amd64.s:198 fp=0x1cc400eadb0 sp=0x1cc400eada8

runtime.mallocgc(0x3ffffffa8, 0x49e020, 0x1, 0xd04008c000)

        /home/newtong/work/go/src/runtime/malloc.go:199 +0x9f3 fp=0x1cc400eae60 sp=0x1cc400eadb0

runtime.newarray(0x49e020, 0x7ffffff5, 0x7f7a3e693000)

        /home/newtong/work/go/src/runtime/malloc.go:365 +0xc1 fp=0x1cc400eae98 sp=0x1cc400eae60

runtime.makeslice(0x499ee0, 0x7ffffff5, 0x7ffffff5, 0x0, 0x0, 0x0)

        /home/newtong/work/go/src/runtime/slice.go:32 +0x15c fp=0x1cc400eaee0 sp=0x1cc400eae98

main.main()

        /home/newtong/work/lavidallocate/go/goAllocTest/main.go:19 +0x19b fp=0x1cc400eaf98 sp=0x1cc400eaee0

runtime.main()

        /home/newtong/work/go/src/runtime/proc.go:63 +0xf3 fp=0x1cc400eafe0 sp=0x1cc400eaf98

runtime.goexit()

        /home/newtong/work/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0x1cc400eafe8 sp=0x1cc400eafe0

75930544

$

With 'top' showing:

Tasks: 1410 total,   1 running, 1409 sleeping,   0 stopped,   0 zombie

Cpu(s):  1.3%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  1058921992k total, 24696588k used, 1034225404k free,    81204k buffers

Swap:  1023992k total,   160088k used,   863904k free,   125920k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                      

67260 newtong  20   0  527g  17g  644 S 110.1  1.7   0:48.50 goAllocTest 

I will be changing to:

malloc.h:  MHeapMap_Bits = 48 - PageShift

(was 40)

and see where it gets me!  :-)

thanks,

Glen

[Dave Cheney]

Why does that variable keep getting set ? Do we just need to delete any reference to it from the documentation and then cop the support load of the small number of cases that do require it ?

[Glen Newton]

I changed to 48 and recompiled:

$ ./all.bash

.

<stuff removed

.

.

# testmain

/lib/../lib64/libpthread.a(libpthread.o): In function `sem_open':

(.text+0x6858): warning: the use of `mktemp' is dangerous, better use `mkstemp'

ok   _/home/newtong/work/go/misc/cgo/testtls 0.004s

ok   _/home/newtong/work/go/misc/cgo/nocgo 0.002s

ok   _/home/newtong/work/go/misc/cgo/nocgo 0.002s

# testmain

/lib/../lib64/libpthread.a(libpthread.o): In function `sem_open':

(.text+0x6858): warning: the use of `mktemp' is dangerous, better use `mkstemp'

ok   _/home/newtong/work/go/misc/cgo/nocgo 0.005s

# Testing race detector.

--- FAIL: TestOutput (0.11s)

output_test.go:54: failed test case simple, expect:

==================

WARNING: DATA RACE

Write by goroutine [0-9]:

 main\.store\(\)

     .+/main\.go:12 \+0x[0-9,a-f]+

 main\.racer\(\)

     .+/main\.go:19 \+0x[0-9,a-f]+

Previous write by main goroutine:

 main\.store\(\)

     .+/main\.go:12 \+0x[0-9,a-f]+

 main\.main\(\)

     .+/main\.go:8 \+0x[0-9,a-f]+

Goroutine [0-9] \(running\) created at:

 main\.startRacer\(\)

     .+/main\.go:15 \+0x[0-9,a-f]+

 main\.main\(\)

     .+/main\.go:7 \+0x[0-9,a-f]+

==================

Found 1 data race\(s\)

exit status 66

got:

FATAL: ThreadSanitizer CHECK failed: ../rtl/tsan_platform.h:119 "((IsShadowMem(shadow))) != (0)" (0x0, 0x0)

exit status 1

FAIL

FAIL runtime/race 0.119s

$

Changed to 46

$ ./all.bash

.

<stuff removed

.

.

k-nucleotide

k-nucleotide-parallel

mandelbrot

meteor-contest

pidigits

threadring

chameneosredux

real 0m4.937s

user 0m4.269s

sys 0m0.672s

# ../test/bench/go1

ok   _/home/newtong/work/go/test/bench/go1 1.680s

# ../test

Where it has sat for the last 20 minutes....

'top' says this:

 16862 newtong   20   0  3492  512  264 R 100.2  0.0  15:58.00 bug273 

Thanks,

Glen                                                                     

[Daniel Theophanes]

You really shouldn't do that.

putting it at 40 allows 1TB of memory. Do you need more? That's where it crashed last time, when you allocated 1TB of memory. I don't think you need 2^48 of memory, and that might indeed cause problems in the runtime.

-Daniel

[Glen Newton]

But when I set it to 40, it failed when I had only allocated 62 2GB
int arrays and 18GB resident and >1TB virtual. With the Java example
program, I was able to allocate 122 2GB int arrays and 1TB resident
before failing.

>I don't think you need 2^48 of memory

OK, Bill Gates ;-)

I _may_ not need 2^48 of memory. But I am not sure what Go needs for
me to do what I need to do, in, say <1TB. Right now I cannot allocate
100 2GB int arrays in Go, even if I set MHeapMap_Bits to 40; I can do
that right now in Java.
In this space (scientific computing / bioinformatics), these kinds of
memory requirements are common.
I would like to use Go to solve some of these problems.
Should I be talking to the Rust people instead? ;-) [I prefer Go]

Thanks,
-Glen

[Dan Kortschak]

amd64 has only 48 address lines AFAIK.

[Dan Kortschak]

BTW If the code you are using is the same as that posted in the OP, you are allocating 2G"int" not 2GB slices (i.e. 16GB slices).

[Glen Newton]

Aaaaah, useful :-)

Thanks!!!

I am trying 43

44 compiler build gets stuck on issue4085b
19832 newtong 20 0 3608 756 380 R 100.1 0.0 1:09.22
issue4085b

[Glen Newton]

Just looked at what I posted: yes, dumb math error!

Thanks for catching this!
-Glen :-)

[Glen Newton]

Building with MHeapMap_Bits = 43 - PageShift  fails:

$ ./all.bash 

# Building C bootstrap tool.

cmd/dist

.

.

<stuff removed>

.

.

k-nucleotide-parallel

mandelbrot

meteor-contest

pidigits

threadring

chameneosredux

real 0m4.903s

user 0m4.226s

sys 0m0.649s

# ../test/bench/go1

ok   _/home/newtong/work/go/test/bench/go1 1.725s

# ../test

# go run run.go -- fixedbugs/issue4085b.go

exit status 1

runtime: out of memory: cannot allocate 8796093022208-byte block (1048576 in use)

fatal error: out of memory

runtime stack:

runtime.throw(0x47b70a)

/home/newtong/work/go/src/runtime/panic.go:491 +0xad

runtime.largeAlloc_m()

/home/newtong/work/go/src/runtime/malloc.c:390 +0xee

runtime.onM(0x4886f0)

/home/newtong/work/go/src/runtime/asm_amd64.s:257 +0x68

runtime.mstart()

/home/newtong/work/go/src/runtime/proc.c:818

goroutine 1 [running]:

runtime.switchtoM()

/home/newtong/work/go/src/runtime/asm_amd64.s:198 fp=0x1420001d5c0 sp=0x1420001d5b8

runtime.mallocgc(0x80000000000, 0x43f9e0, 0x1, 0x1420000a100)

/home/newtong/work/go/src/runtime/malloc.go:199 +0x9f3 fp=0x1420001d670 sp=0x1420001d5c0

runtime.newarray(0x43f9e0, 0x10000000000, 0x442e60)

/home/newtong/work/go/src/runtime/malloc.go:365 +0xc1 fp=0x1420001d6a8 sp=0x1420001d670

runtime.makeslice(0x440260, 0x10000000000, 0x10000000000, 0x0, 0x0, 0x0)

/home/newtong/work/go/src/runtime/slice.go:32 +0x15c fp=0x1420001d6f0 sp=0x1420001d6a8

main.func·003()

/home/newtong/work/go/test/fixedbugs/issue4085b.go:24 +0x3e fp=0x1420001d740 sp=0x1420001d6f0

main.shouldPanic(0x457710, 0x10, 0x1420001d780)

/home/newtong/work/go/test/fixedbugs/issue4085b.go:48 +0x5d fp=0x1420001d760 sp=0x1420001d740

main.main()

/home/newtong/work/go/test/fixedbugs/issue4085b.go:24 +0xdc fp=0x1420001d798 sp=0x1420001d760

runtime.main()

/home/newtong/work/go/src/runtime/proc.go:63 +0xf3 fp=0x1420001d7e0 sp=0x1420001d798

runtime.goexit()

/home/newtong/work/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0x1420001d7e8 sp=0x1420001d7e0

goroutine 2 [runnable]:

runtime.forcegchelper()

/home/newtong/work/go/src/runtime/proc.go:90

runtime.goexit()

/home/newtong/work/go/src/runtime/asm_amd64.s:2232 +0x1

goroutine 3 [runnable]:

runtime.bgsweep()

/home/newtong/work/go/src/runtime/mgc0.go:82

runtime.goexit()

/home/newtong/work/go/src/runtime/asm_amd64.s:2232 +0x1

exit status 2

FAIL fixedbugs/issue4085b.go 0.111s

real 0m12.753s

user 1m1.712s

sys 0m13.648s

[Dan Kortschak]

I regularly use bigger heaps (also for bioinfo things) without issue. Perhaps try using a value that you have physical backing for?

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.

To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

[Glen Newton]

Bigger heaps.... in Java? I am OK with the Java side of things. I am
trying to solve the Go side.
Perhaps I am not understanding you?

Thanks,
Glen

On Tue, Dec 23, 2014 at 4:53 PM, Dan Kortschak

[Dan Kortschak]

In Go, using the method described by Daniel Theophanes.

[Glen Newton]

Hi Dan,

I tried to go to:

MHeapMap_Bits = 43 - PageShift

but the build fails as described above.

I was able to build with
MHeapMap_Bits = 42 - PageShift

With this, I was able to allocate 254 16GB int arrays!

Thanks,
Glen

[Daniel Theophanes]

62 chucks x 16 GiB = 992 GiB

Go is doing just fine.

-Daniel

[Chris Kastorff]

Since we're already in rocky territory, consider allocating space for
your data yourself (e.g. using the mmap syscall). You can turn that
space into a typed array of whatever you like and slice it as you desire.

Cgo-like restrictions apply here; you must free it yourself at an
appropriate time, and you must not use pointers to objects allocated by
the Go runtime (in stack or in heap) that are in your custom memory
location (the GC won't see those pointers nor will it update them if it
decides to relocate some data.) Pointers from the special space back
into that special space are fine, though.

If your data structures are simple enough, this might work, but if you
want the power of the garbage collector, it's probably more pain than
modifying the runtime itself.

-Chris K

> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com

> <mailto:golang-nuts...@googlegroups.com>.

[Glen Newton]

>I don't think you need 2^48 of memory

This article: http://www.theplatform.net/2015/03/03/dna-sequencing-not-quite-hpc-yet/
describes: "Samtools sort of 1.38 TB of aligned reads..."
samtools is a genomics suite of tools for DNA sequence alignment: http://en.wikipedia.org/wiki/SAMtools
While samtools does have an option to sort in chunks (it sorts in-memory) then put these into temp files to be assembled at the end, sorting in-memory is much faster, so 2^48 of memory is not that crazy for some compute domains...  :-)

-Glen