I run this program on one of our new huge servers (1 TB ram, 40 core (10core x 4 cpu) Xeon E5-4650 v2, CentOS release 6.5, go version go1.4 linux/amd64 ):

First, yes, you are hitting the Go VM limit for 64-bit programs, which
is a bit underneath 128GB per process. A Go developer may be able to
explain why that limit was chosen and if it's tunable; I don't know.

Second, to explain why it works the same without physical ram to back it:
You're allocating space, but you're never using any of it. The kernel's
virtual memory manager never gets a request to write to almost all of
the pages you're creating, so it doesn't bother mapping a physical page
to the virtual address space in the program, so you don't run out of
physical pages.
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com
> <mailto:golang-nuts+unsubscribe@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

There is a blog post http://utcc.utoronto.ca/~cks/space/blog/programming/GoBigVirtualSize on the subject.

On Tuesday, December 23, 2014 8:11:36 AM UTC-8, Glen Newton wrote:

Hello,

I run this program on one of our new huge servers (1 TB ram, 40 core (10core x 4 cpu) Xeon E5-4650 v2, CentOS release 6.5, go version go1.4 linux/amd64 ):

https://play.golang.org/p/yAJPIu-WSf

[snip]                                     

Both fail close to 127GB virtual: is this a magic number wrt the memory model?

Any help in getting me to understand this would be appreciated.

Thanks,

Glen

I would recommend you read the following from the go source file:

Thank-you all for your responses: I understand this behaviour now.

Setting a constant and recompiling is neither hard nor hacking. Be bold.

Hi Glen,

I followed your instructions, and recompiled the application. On my laptop, getting:

Doh!

Why does that variable keep getting set ? Do we just need to delete any reference to it from the documentation and then cop the support load of the small number of cases that do require it ?

I changed to 48 and recompiled:

You really shouldn't do that.

But when I set it to 40, it failed when I had only allocated 62 2GB
int arrays and 18GB resident and >1TB virtual. With the Java example
program, I was able to allocate 122 2GB int arrays and 1TB resident
before failing.

>I don't think you need 2^48 of memory

OK, Bill Gates  ;-)

I _may_ not need 2^48 of memory. But I am not sure what Go needs for
me to do what I need to do, in, say <1TB. Right now I cannot allocate
100 2GB int arrays in Go, even if I set MHeapMap_Bits to 40; I can do
that right now in Java.
In this space (scientific computing / bioinformatics), these kinds of
memory requirements are common.
I would like to use Go to solve some of these problems.
  Should I be talking to the Rust people instead?  ;-)     [I prefer Go]

Thanks,
-Glen

amd64 has only 48 address lines AFAIK.

BTW If the code you are using is the same as that posted in the OP, you are allocating 2G"int" not 2GB slices (i.e. 16GB slices).

Aaaaah, useful  :-)

Thanks!!!

I am trying 43

44 compiler build gets stuck on issue4085b
19832 newtong   20   0  3608  756  380 R 100.1  0.0   1:09.22
issue4085b

Just looked at what I posted: yes, dumb math error!

Thanks for catching this!
-Glen :-)

Building with MHeapMap_Bits = 43 - PageShift  fails:

Bigger heaps.... in Java? I am OK with the Java side of things. I am
trying to solve the Go side.
 Perhaps I am not understanding you?

Thanks,
Glen

On Tue, Dec 23, 2014 at 4:53 PM, Dan Kortschak

In Go, using the method described by Daniel Theophanes.

Hi Dan,

I tried to go to:

  MHeapMap_Bits = 43 - PageShift

but the build fails as described above.

I was able to build with
 MHeapMap_Bits = 42 - PageShift

With this, I was able to allocate 254 16GB int arrays!

Thanks,
Glen

62 chucks x 16 GiB = 992 GiB

Go is doing just fine.

-Daniel

Since we're already in rocky territory, consider allocating space for
your data yourself (e.g. using the mmap syscall). You can turn that
space into a typed array of whatever you like and slice it as you desire.

Cgo-like restrictions apply here; you must free it yourself at an
appropriate time, and you must not use pointers to objects allocated by
the Go runtime (in stack or in heap) that are in your custom memory
location (the GC won't see those pointers nor will it update them if it
decides to relocate some data.) Pointers from the special space back
into that special space are fine, though.

If your data structures are simple enough, this might work, but if you
want the power of the garbage collector, it's probably more pain than
modifying the runtime itself.

    -Chris K

On 12/23/2014 10:49 AM, Glen Newton wrote:
> Thank-you all for your responses: I understand this behaviour now.
>
> So a follow-up: we have applications (C++, Java, etc) that need several
> hundred GB to operate (big bioinformatics apps). If I want to implement
> these types of apps in Go, and the underlying algorithm is similar
> (needing this kind of space), how should I proceed if I am going to
> need, say 1/2 TB of ram?
> It seems that I a locked out of writing Go programs that need this kind
> of memory. I would prefer this was not the case. :-(
>
> Is there a time frame for when this limit will not be the default in Go?
> [I would prefer not to hack the Go compiler with unintended consequences...]
>
> ---
>
> Below is a similar similar Java application that also allocates in a
> similar fashion. I wrote just as a reality check.
> It creates 121 int[] before OOM. Just before it fails, 'top' says it is
> 1TB resident:
>
> Tasks: 1401 total,   1 running, 1400 sleeping,   0 stopped,   0 zombie
> Cpu(s): 58.2%us,  1.3%sy,  0.0%ni, 40.5%id,  0.0%wa,  0.0%hi,  0.0%si,
>  0.0%st
> Mem:  1058921992k total, 1056937768k used,  1984224k free,    18256k buffers
> Swap:  1023992k total,   165292k used,   858700k free,    85812k cached
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 21468 newtong  20   0     ? 1.0t 8080 S 4771.9 99.0  15:34.55 java  
>
>
> public class AllocTest{
>     public static final void main(String[] args){
>         AllocTest at = new AllocTest();
>         at.start();
>     }
>     public void start(){
>         int max = Integer.MAX_VALUE-10;
>         System.out.println(max);
>         int[][] z = new int[max][];
>    
>         // Testing showed OOM at sub~=121                              
>                                                                
>         int sub = 130;
>         for(int i=0; i<sub; i++){
>             System.out.println(max + " " + i);
>             try{
>                 z[i] = new int[max];
>             }catch(Throwable t){
>                 t.printStackTrace();
>                 return;
>             }
>         }
>     }
> }
>
> Thanks,
> Glen
>
> On Tuesday, December 23, 2014 1:06:00 PM UTC-5, Daniel Theophanes wrote:
>
>     I would recommend you read the following from the go source file:
>     https://github.com/golang/go/blob/release-branch.go1.4/src/runtime/malloc.h#L152
>     <https://github.com/golang/go/blob/release-branch.go1.4/src/runtime/malloc.h#L152>
>
>     Adjusting the constants above the linked line should change your max
>     limits. I *think* you can safely adjust it to other sane values.
>
>     -Daniel
>
>
>     On Tuesday, December 23, 2014 8:11:36 AM UTC-8, Glen Newton wrote:
>
>         Hello,
>
>         I run this program on one of our new huge servers (1 TB ram, 40
>         core (10core x 4 cpu) Xeon E5-4650 v2, CentOS release 6.5, go
>         version go1.4 linux/amd64 ):
>
>         https://play.golang.org/p/yAJPIu-WSf
>         <https://play.golang.org/p/yAJPIu-WSf>
>
>         $ /usr/bin/time -f "%M" ./goAllocTest
>         0
>         1
>         2
>         3
>         4
>         5
>         6
>         7
>         8
>         9
>         10
>         11
>         12
>         13
>         14
>         runtime: out of memory: cannot allocate 8589934592-byte block
>         (128850067456 in use)
>         fatal error: out of memory
>
>         runtime stack:
>         runtime.throw(0x545e2a)
>                 /usr/local/go/src/runtime/panic.go:491 +0xad
>         runtime.largeAlloc_m()
>                 /usr/local/go/src/runtime/malloc.c:390 +0xee
>         runtime.onM(0x54bab0)
>                 /usr/local/go/src/runtime/asm_amd64.s:257 +0x68
>         runtime.mstart()
>                 /usr/local/go/src/runtime/proc.c:818
>
>         goroutine 1 [running]:
>         runtime.switchtoM()
>                 /usr/local/go/src/runtime/asm_amd64.s:198
>         fp=0xe0080e0db0 sp=0xe0080e0da8
>         runtime.mallocgc(0x1ffffffd0, 0x49e020, 0x1, 0x54be40)
>                 /usr/local/go/src/runtime/malloc.go:199 +0x9f3
>         fp=0xe0080e0e60 sp=0xe0080e0db0
>         runtime.newarray(0x49e020, 0x3ffffffa, 0x7f9d7adb0000)
>                 /usr/local/go/src/runtime/malloc.go:365 +0xc1
>         fp=0xe0080e0e98 sp=0xe0080e0e60
>         runtime.makeslice(0x499ee0, 0x3ffffffa, 0x3ffffffa, 0x0, 0x0, 0x0)
>                 /usr/local/go/src/runtime/slice.go:32 +0x15c
>         fp=0xe0080e0ee0 sp=0xe0080e0e98
>         main.main()
>                
>         /home/newtong/work/lavidallocate/go/goAllocTest/main.go:19
>         +0x19b fp=0xe0080e0f98 sp=0xe0080e0ee0
>         runtime.main()
>                 /usr/local/go/src/runtime/proc.go:63 +0xf3
>         fp=0xe0080e0fe0 sp=0xe0080e0f98
>         runtime.goexit()
>                 /usr/local/go/src/runtime/asm_amd64.s:2232 +0x1
>         fp=0xe0080e0fe8 sp=0xe0080e0fe0
>         36239904
>
>         'top' indicates that this process had 127GB virtual, 8.6GB
>         resident just before it failed. The above /usr/bin/time
>         indicates 36GB maximum resident.
>          
>         I don't understand why this failed to early: I would have
>         expected it to continue allocating a lot longer. 'top' at
>         count=13 indicated:
>         Mem:  1058921992k total, 20655084k used, 1038266908k free,  
>         310636k buffers
>         15705 newtong  20   0  127g 8.6g  644 S 106.5  0.9   0:15.81
>         goAllocTest                                                    
>                    
>
>         So it looks to me that there is a lot of memory available.
>
>         Is there a problem with the above code? My calculations is that
>         the initial alloc of for the 'var a []*[]int' is 2147483637*64b
>         (64 bit pointer, correct?) = 2147483637*8B = 16GB; then each
>         subsequent alloc for the '[]int' is 2147483637*32b
>         = 2147483637*4B=8GB.
>         So initial alloc of 'a' plus 14 allocs of []int = 32GB + 14*8GB
>         = 144GB
>
>         When I run this on my laptop with only 16GB ram (Fedora18), I
>         get the same issue _at_the_same_point (14 allocs). This suggests
>         that either my code is wrong and I am not understanding things
>         or....?
>         'top' just before failing on laptop:
>         KiB Mem:  16373032 total,  5681960 used, 10691072 free,   508400
>         buffers
>
>          5998 newtong   20   0  119g 1.1g  644 S 105.1  7.2   0:20.82
>         goAllocTest                                                    
>                    
>         Both fail close to 127GB virtual: is this a magic number wrt the
>         memory model?
>         Any help in getting me to understand this would be appreciated.
>
>         Thanks,
>         Glen
>
>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com
> <mailto:golang-nuts+unsubscribe@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

>I don't think you need 2^48 of memory

This article: http://www.theplatform.net/2015/03/03/dna-sequencing-not-quite-hpc-yet/
describes: "Samtools sort of 1.38 TB of aligned reads..."
samtools is a genomics suite of tools for DNA sequence alignment: http://en.wikipedia.org/wiki/SAMtools
While samtools does have an option to sort in chunks (it sorts in-memory) then put these into temp files to be assembled at the end, sorting in-memory is much faster, so 2^48 of memory is not that crazy for some compute domains...  :-)

-Glen

On Tuesday, December 23, 2014 at 3:52:44 PM UTC-5, Daniel Theophanes wrote:

You really shouldn't do that.

putting it at 40 allows 1TB of memory. Do you need more? That's where it crashed last time, when you allocated 1TB of memory. I don't think you need 2^48 of memory, and that might indeed cause problems in the runtime.

-Daniel

On Tue, Dec 23, 2014 at 12:49 PM, Glen Newton <glen....@gmail.com> wrote:

I changed to 48 and recompiled:

$ ./all.bash

.

<stuff removed

.

.

# testmain

/lib/../lib64/libpthread.a(libpthread.o): In function `sem_open':

(.text+0x6858): warning: the use of `mktemp' is dangerous, better use `mkstemp'

ok   _/home/newtong/work/go/misc/cgo/testtls 0.004s

ok   _/home/newtong/work/go/misc/cgo/nocgo 0.002s

ok   _/home/newtong/work/go/misc/cgo/nocgo 0.002s

# testmain

/lib/../lib64/libpthread.a(libpthread.o): In function `sem_open':

(.text+0x6858): warning: the use of `mktemp' is dangerous, better use `mkstemp'

ok   _/home/newtong/work/go/misc/cgo/nocgo 0.005s

# Testing race detector.

--- FAIL: TestOutput (0.11s)

output_test.go:54: failed test case simple, expect:

==================

WARNING: DATA RACE

Write by goroutine [0-9]:

 main\.store\(\)

     .+/main\.go:12 \+0x[0-9,a-f]+

 main\.racer\(\)

     .+/main\.go:19 \+0x[0-9,a-f]+

Previous write by main goroutine:

 main\.store\(\)

     .+/main\.go:12 \+0x[0-9,a-f]+

 main\.main\(\)

     .+/main\.go:8 \+0x[0-9,a-f]+

Goroutine [0-9] \(running\) created at:

 main\.startRacer\(\)

     .+/main\.go:15 \+0x[0-9,a-f]+

 main\.main\(\)

     .+/main\.go:7 \+0x[0-9,a-f]+

==================

Found 1 data race\(s\)

exit status 66

got:

FATAL: ThreadSanitizer CHECK failed: ../rtl/tsan_platform.h:119 "((IsShadowMem(shadow))) != (0)" (0x0, 0x0)

exit status 1

FAIL

FAIL runtime/race 0.119s

$

Changed to 46

$ ./all.bash

.

<stuff removed

.

.

k-nucleotide

k-nucleotide-parallel

mandelbrot

meteor-contest

pidigits

threadring

chameneosredux

real 0m4.937s

user 0m4.269s

sys 0m0.672s

# ../test/bench/go1

ok   _/home/newtong/work/go/test/bench/go1 1.680s

# ../test

Where it has sat for the last 20 minutes....

'top' says this:

 16862 newtong   20   0  3492  512  264 R 100.2  0.0  15:58.00 bug273 

Thanks,

Glen                                                                     

On Tuesday, December 23, 2014 3:44:15 PM UTC-5, Dave Cheney wrote:

Why does that variable keep getting set ? Do we just need to delete any reference to it from the documentation and then cop the support load of the small number of cases that do require it ?

On Wednesday, 24 December 2014 07:25:09 UTC+11, Glen Newton wrote:

Doh!

Yes, that was it.

Now on the 1TB server I am getting:

$ /usr/bin/time -f "%M" ./goAllocTest

0

1

2

.

.<stuff removed>...

.

60

61

62

runtime: out of memory: cannot allocate 17179869184-byte block (1082332807168 in use)

fatal error: out of memory

runtime stack:

runtime.throw(0x545e2a)

        /home/newtong/work/go/src/runtime/panic.go:491 +0xad

runtime.largeAlloc_m()

        /home/newtong/work/go/src/runtime/malloc.c:390 +0xee

runtime.onM(0xd040014400)

        /home/newtong/work/go/src/runtime/asm_amd64.s:257 +0x68

runtime.mstart()

        /home/newtong/work/go/src/runtime/proc.c:818

goroutine 1 [running]:

runtime.switchtoM()

        /home/newtong/work/go/src/runtime/asm_amd64.s:198 fp=0x1cc400eadb0 sp=0x1cc400eada8

runtime.mallocgc(0x3ffffffa8, 0x49e020, 0x1, 0xd04008c000)

        /home/newtong/work/go/src/runtime/malloc.go:199 +0x9f3 fp=0x1cc400eae60 sp=0x1cc400eadb0

runtime.newarray(0x49e020, 0x7ffffff5, 0x7f7a3e693000)

        /home/newtong/work/go/src/runtime/malloc.go:365 +0xc1 fp=0x1cc400eae98 sp=0x1cc400eae60

runtime.makeslice(0x499ee0, 0x7ffffff5, 0x7ffffff5, 0x0, 0x0, 0x0)

        /home/newtong/work/go/src/runtime/slice.go:32 +0x15c fp=0x1cc400eaee0 sp=0x1cc400eae98

main.main()

        /home/newtong/work/lavidallocate/go/goAllocTest/main.go:19 +0x19b fp=0x1cc400eaf98 sp=0x1cc400eaee0

runtime.main()

        /home/newtong/work/go/src/runtime/proc.go:63 +0xf3 fp=0x1cc400eafe0 sp=0x1cc400eaf98

runtime.goexit()

        /home/newtong/work/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0x1cc400eafe8 sp=0x1cc400eafe0

75930544

$

With 'top' showing:

Tasks: 1410 total,   1 running, 1409 sleeping,   0 stopped,   0 zombie

Cpu(s):  1.3%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:  1058921992k total, 24696588k used, 1034225404k free,    81204k buffers

Swap:  1023992k total,   160088k used,   863904k free,   125920k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                      

67260 newtong  20   0  527g  17g  644 S 110.1  1.7   0:48.50 goAllocTest 

I will be changing to:

malloc.h:  MHeapMap_Bits = 48 - PageShift

(was 40)

and see where it gets me!  :-)

thanks,

Glen

On Tuesday, December 23, 2014 3:12:59 PM UTC-5, Dave Cheney wrote:

The stack trace shows remnants of your old Go install in /usr/local/go. Did you have GOROOT set ?

On Wednesday, 24 December 2014 06:31:34 UTC+11, Glen Newton wrote:

I followed your instructions, and recompiled the application. On my laptop, getting:

$ ./all.bash

# Building C bootstrap tool.

.

. <stuff removed>...

.

Installed Go for linux/amd64 in /home/newtong/work/go

Installed commands in /home/newtong/work/go/bin

*** You need to add /home/newtong/work/go/bin to your PATH.

$ export PATH=/home/newtong/work/go/bin:$PATH

$ which go

~/work/go/bin/go

$ cd ~/work/goAllocTest

$ rm goAllocTest

$ go build

$ /usr/bin/time -f "%M" ./goAllocTest 

0

1

2

3

4

5

6

runtime: out of memory: cannot allocate 17179869184-byte block (120260132864 in use)

fatal error: out of memory

runtime stack:

runtime.throw(0x545e2a)

/usr/local/go/src/runtime/panic.go:491 +0xad

runtime.largeAlloc_m()

/usr/local/go/src/runtime/malloc.c:390 +0xee

runtime.onM(0x54bab0)

/usr/local/go/src/runtime/asm_amd64.s:257 +0x68

runtime.mstart()

/usr/local/go/src/runtime/proc.c:818

goroutine 1 [running]:

runtime.switchtoM()

/usr/local/go/src/runtime/asm_amd64.s:198 fp=0xde0809cdb0 sp=0xde0809cda8

runtime.mallocgc(0x3ffffffa8, 0x49e020, 0x1, 0x54be40)

/usr/local/go/src/runtime/malloc.go:199 +0x9f3 fp=0xde0809ce60 sp=0xde0809cdb0

runtime.newarray(0x49e020, 0x7ffffff5, 0x7f4e81da3000)

/usr/local/go/src/runtime/malloc.go:365 +0xc1 fp=0xde0809ce98 sp=0xde0809ce60

runtime.makeslice(0x499ee0, 0x7ffffff5, 0x7ffffff5, 0x0, 0x0, 0x0)

/usr/local/go/src/runtime/slice.go:32 +0x15c fp=0xde0809cee0 sp=0xde0809ce98

main.main()

/home/newtong/work/lavidallocate/go/goAllocTest/main.go:19 +0x19b fp=0xde0809cf98 sp=0xde0809cee0

runtime.main()

/usr/local/go/src/runtime/proc.go:63 +0xf3 fp=0xde0809cfe0 sp=0xde0809cf98

runtime.goexit()

/usr/local/go/src/runtime/asm_amd64.s:2232 +0x1 fp=0xde0809cfe8 sp=0xde0809cfe0

1174816

Suggestions?

Thanks,

Glen

On Tuesday, December 23, 2014 1:55:32 PM UTC-5, Daniel Theophanes wrote:

Hi Glen,

Go can handle it.

Pull the git source and update to 1.4

git clone https://github.com/golang/go

git checkout go1.4

git clean -f

In file: src/runtime/malloc.h

Change line 135 to:

MHeapMap_Bits = 40 - PageShift,

cd src

./all.bash

-Daniel

Glen

Adjusting the constants above the linked line should change your max limits. I *think* you can safely adjust it to other sane values.

-Daniel

On Tuesday, December 23, 2014 8:11:36 AM UTC-8, Glen Newton wrote:

Hello,

I run this program on one of our new huge servers (1 TB ram, 40 core (10core x 4 cpu) Xeon E5-4650 v2, CentOS release 6.5, go version go1.4 linux/amd64 ):

https://play.golang.org/p/yAJPIu-WSf

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/Ij1uf9ql8Zk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/Ij1uf9ql8Zk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.