Virt memory consumption

[Xan]

hi,

I run this code:

package main
.
import (
"fmt"
"http"
)
.
func main() {
http.HandleFunc("/",
func(w http.ResponseWriter, req *http.Request) {
fmt.Fprintln(w, "Goodbye, World!")
})
if err := http.ListenAndServe(":8080", nil); err != nil {
fmt.Println(err)
}
}



and when I compiled and I run ./webserver.go I see in htop that I
have:


1 [||||||
7.2%] Tasks: 81, 131 thr; 1 running
2 [|||||||
9.7%] Load average: 0.05 0.11 0.19
Mem[||||||||||||||||||||||||||||||||||||
204/992MB] Uptime: 04:49:10
Swp[||
220/9536MB]

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
2971 xan 20 0 794M 2288 1696 S 0.0 0.2 0:00.02 ./
webserver.out
2972 xan 20 0 794M 2288 1696 S 0.0 0.2 0:00.02 ./
webserver.out
3029 xan 20 0 794M 2288 1696 S 0.0 0.2 0:00.00 ./
webserver.out

when it's more and more virtual memory. Why Go consumes so much
virtual memory while the res and shr are so small?

Thanks a lot,

[minux]

On Thu, Mar 22, 2012 at 3:17 AM, Xan <xanc...@gmail.com> wrote:

 PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 2971 xan       20   0  794M  2288  1696 S  0.0  0.2  0:00.02 ./
webserver.out
 2972 xan       20   0  794M  2288  1696 S  0.0  0.2  0:00.02 ./
webserver.out
 3029 xan       20   0  794M  2288  1696 S  0.0  0.2  0:00.00 ./
webserver.out

when it's more and more virtual memory. Why Go consumes so much
virtual memory while the res and shr are so small?

This relates to the way Go's runtime works. It reserves large chunk of virtual memory space for its latter use.

And you don't need to worry about it, because it is "virtual", and doesn't indicate the amount of physical

memory used (read RES if you want to know the actual memory usage).

[Xan]

Okay, but Virtual uses RAM + Swap, isn't? So what happens if we have
less ram+swap left?

Thanks,


On Mar 21, 8:22 pm, minux <minux...@gmail.com> wrote:

[DisposaBoy]

No, it's virtual(not real)... which means it doesn't necessarily, and usually doesn't use anywhere near that amount. Your best option is to simply ignore it altogether

[Xan]

@DisposaBoy: so:
1) what is the limit of virtual memory?
2) which algorithm use Go for reserve virt memory?

Thanks,
Xan.

[Rémy Oudompheng]

Le 21 mars 2012 21:27, Xan <xanc...@gmail.com> a écrit :
> @DisposaBoy: so:
> 1) what is the limit of virtual memory?

It is usually a configuration parameter of your system. In 32 bits, it
is probably 3GB and can't get above. In 64-bit, there might be a
limit, and otherwise there is virtually none (here I think it's
256TB).

> 2) which algorithm use Go for reserve virt memory?

In 32 bits, the algorithm consists in reseving 768MB. In 64-bit mode,
it's more complicated, and it's not 768MB from the beginning.

Rémy.

[Xan xan]

2012/3/21 Rémy Oudompheng <remyoud...@gmail.com>

Le 21 mars 2012 21:27, Xan <xanc...@gmail.com> a écrit :
> @DisposaBoy: so:
> 1) what is the limit of virtual memory?

It is usually a configuration parameter of your system. In 32 bits, it
is probably 3GB and can't get above. In 64-bit, there might be a
limit, and otherwise there is virtually none (here I think it's
256TB).

OK 

> 2) which algorithm use Go for reserve virt memory?

In 32 bits, the algorithm consists in reseving 768MB. In 64-bit mode,
it's more complicated, and it's not 768MB from the beginning.

Is there source code of that? Just curious!

Thanks. 

Rémy.

[minux]

On Thu, Mar 22, 2012 at 9:17 PM, Xan xan <xanc...@gmail.com> wrote:

2012/3/21 Rémy Oudompheng <remyoud...@gmail.com>

> 2) which algorithm use Go for reserve virt memory?

In 32 bits, the algorithm consists in reseving 768MB. In 64-bit mode,
it's more complicated, and it's not 768MB from the beginning.

Is there source code of that? Just curious!

http://tip.golang.org/src/pkg/runtime/malloc.goc

Warning: you digging into Go runtime now. This file is written in a mix of Go and C (hence the extension goc).

[krolaw]

Hi,

VM usage can make quite a difference, at least in my case.  I've written a tiny sip redirect (and message forwarding) server, which I host on a VPS with no swap.  Although, rss is under 2.5MB, vsz is 76MB.  Using free, before and after running my app, indicates that it uses 70MB to run, which seems high for a stateless server.  And since, it's running on a VPS with 128MB ram (256 burst), it looks severely disproportionate compared with the other simple services running.

On the flip side, using Go, made it really easy to write and it performs really well.  It's my first system app and I'm very pleased with it.  But if there is a practical way to reduce the ram usage, I'd like to know :-)

Cheers.

[Aram Hăvărneanu]

> Although, rss is under 2.5MB, vsz is 76MB.

Don't worry about it, you have a 4GB address space, of which you can
use 3G, it has nothing to do with your available RAM.

--
Aram Hăvărneanu

[krolaw]

Well, as I said before, I conducted tests using the unix "free" command.  I've included the tests below.  From cold start SipCloak uses 62MB of ram....which coincides with the ps command, also included below.

root@alpha:~# killall SipCloak4

root@alpha:~# free

             total       used       free     shared    buffers     cached

Mem:        262144      17436     244708          0          0          0

-/+ buffers/cache:      17436     244708

Swap:            0          0          0

root@alpha:~# sudo -b -u nobody ~krolaw/SipCloak4

root@alpha:~# free

             total       used       free     shared    buffers     cached

Mem:        262144      70056     192088          0          0          0

-/+ buffers/cache:      70056     192088

Swap:            0          0          0

root@alpha:~# ps up 1382

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

nobody    1382  0.0  0.8  59680  2276 pts/0    Sl   11:39   0:00 /home/krolaw/SipCloak4

Cheers.

[Kyle Lemons]

I think his point was this: Virtual memory is free, occupies no space, and has no performance impact.  Slightly oversimplified, but for most intents and purposes it works.

[krolaw]

Ok, I guess what I am looking for is an explanation as to why, my tiny (practically stateless) go program uses up so much of my VPS' ram.  There also appears to be a direct correlation between the virtual memory my app uses (plus res) and the ram lost.

Are you saying that this correlation is purely coincidence (VM being free)?  Or that my situation is insignificant (not most intents and purposes)?

Cheers.

[Matt Kane's Brain]

I don't have the answer but I think the following is important:
1.) Any code you can show
2.) What imports you are using
3.) 32-bit or 64-bit?

--
matt kane's brain
http://hydrogenproject.com

[Bolang]

On 04/11/2012 07:38 AM, krolaw wrote:
> Ok, I guess what I am looking for is an explanation as to why, my tiny
> (practically stateless) go program uses up so much of my VPS' ram. There
> also appears to be a direct correlation between the virtual memory my
> app uses (plus res) and the ram lost.

I guess your VPS use openvz as virtualization.
AFAIK, openvz VM works different than real Linux. Actually, openvz
is more like containter than virtualization.
But, sorry, i don't understand the detail.

Maybe you can compare with hardware or paravirt virtualization like
xen, vmwavre, kvm, or virtualbox

[krolaw]

Hi Bo,

I think you're right.  I did the same test on my Linux laptop and the free memory only dropped by 2MB.  So VPSes (at least my one) works differently.  In which case, a lot of VPS' would probably benefit from reducing the virtual somehow...including mine.    

I wonder if this problem may also arise on small devices that don't have swap?

BTW, I'm using 64bit Go.  I've been compiling on my laptop and uploading to the VPS, so both are using exactly the same program.

So does anyone know if there's a practical way to reduce the VM of Go apps?

Cheers.

[David Symonds]

On Wed, Apr 11, 2012 at 11:37 AM, krolaw <kro...@gmail.com> wrote:

> So does anyone know if there's a practical way to reduce the VM of Go apps?

There's some hacks, but there's really no point. Virtual memory is
(nearly) free, so there's no point limiting it.

Dave.

[Paul Borman]

Is the problem here that your VPS is counting virtual address space rather than physical memory used when doing its computations?  If so then I would use my finger to point to that as the problem.  As David says, allocated but unused virtual memory is nearly free.

"I'll gladly pay you today for maybe eating a hamburger on Tuesday."

    -Paul

[Dmitry Chestnykh]

On Wednesday, April 11, 2012 3:05:52 AM UTC+2, Bo Lang wrote:

On 04/11/2012 07:38 AM, krolaw wrote:
> Ok, I guess what I am looking for is an explanation as to why, my tiny
> (practically stateless) go program uses up so much of my VPS' ram. There
> also appears to be a direct correlation between the virtual memory my
> app uses (plus res) and the ram lost.

I guess your VPS use openvz as virtualization.
AFAIK, openvz VM works different than real Linux. Actually, openvz
is more like containter than virtualization.

This seems to be it. OpenVZ memory accounting is broken:

1. http://hostingfu.com/article/xen-or-openvz

2. http://hostingfu.com/article/vzfree-checking-memory-usage-inside-openvz-ve

-Dmitry

[Richard Warburton]

Indeed, well not broken, just different... and OpenVZ because of it's simplicity and greater performance is preferred (vs Xen) among VPS hosts, especially the cheaper ones...but it does count virtual memory against you.

As such, I'm limited to one go program running per VPS unless I can reduce the virtual memory.  It isn't such a big deal, but it does severely reduce go's usefulness in such environments, which seem to becoming more common.

Cheers.

But OpenVZ seems to be extremely common on VPSes.

[Kyle Lemons]

On Wed, Apr 11, 2012 at 2:30 PM, Richard Warburton <kro...@gmail.com> wrote:

Indeed, well not broken, just different... and OpenVZ because of it's simplicity and greater performance

Is greater performance worth it if they're cutting these sorts of corners? Seems dangerous to me.

[Dave Cheney]

Hello,

I have hosted lots of Go programs inside OpenVZ without incident, the
issues with OpenVZ generally relate to running Java processes inside
OpenVZ containers. I don't think OpenVZ was the OP's problem, or if in
fact there was a problem, my reading of the OP's post was they were
confused by the values reported in top and free, not that their VPS
was unable to cope with the 768mb VSS footprint on 32bit linux.

Cheers

Dave

[Richard Warburton]

On a VPS with 768MB ram or larger, it's probably a non-issue.

But you can do a lot with just 128MB VPS or 256MB VPS unless you are running Go or Java.  And with 128MB VPS' now as low as $12/year, almost anyone has the funds to experiment with running their own server - and there's heaps of mail (sendmail), dns (nsd), web (nginx), that will run beautifully on these systems in low ram.

Now you can argue than OpenVZ just isn't cool, and you'd be right.  But it is the most common VPS option out there, and the only option at really low prices.

You can argue that I'm just being cheap and if I'm serious about go, I should just front up serious cash, and you would be right again...

Cheers.

P.S. Does anyone know if BSD Jail based VPSes count VM?

--
Richard Warburton - MSc(Hons), PGDipSci, BE(Hons) - http://richard.warburton.it/

GMT21-9 Skype/GTalk:krolaw NZ+6499735099 US+14247574049 UK+448452872990

P.O. Box 41378, St Lukes, Auckland 1346, New Zealand

[Andy W. Song]

Maybe Dave is right, it could be that just your top/free command under OpenVZ is broken. Try check memory usage under your VPS console.

Andy

--

---------------------------------------------------------------

有志者，事竟成，破釜沉舟，百二秦关终属楚

苦心人，天不负，卧薪尝胆，三千越甲可吞吴

[Richard Warburton]

Console also confirms usage :-)

Hopefully, these quotes with links will help:

"With OpenVZ/Virtuozzo the memory is measured in terms of virtual memory pages allocated. You might hit the memory limit sooner than expected when programs allocate a lot of memory that it not really used."

http://www.moeding.net/archives/16-XEN-and-OpenVZ-Differences-with-memory-limits.html

"There are a few problems and a few advantages with OpenVZ’s approach to memory management. One of the biggest problem is the amount of memory an application uses and the amount of memory an application allocates is actually different, and the difference can vary a lot depending on the application. Take Java for example, it usually allocates a huge chunk of memory — usually everything it can see the host node has — but it might only use/commit a small fraction of allocated memory. It can usually render a Java program unusable as you will pretty much hit the privvmpages limit straight away."

http://hostingfu.com/article/xen-or-openvz

--

[Richard Warburton]

http://www.moeding.net/archives/20-Optimizing-virtual-memory-in-OpenVZ-I.html

Looks like other apps, know about this and provide facilities to compensate.

Cheers.

[minux]

On Thu, Apr 12, 2012 at 5:30 AM, Richard Warburton <kro...@gmail.com> wrote:

Indeed, well not broken, just different... and OpenVZ because of it's simplicity and greater performance is preferred (vs Xen) among VPS hosts, especially the cheaper ones...but it does count virtual memory against you.

As such, I'm limited to one go program running per VPS unless I can reduce the virtual memory.  It isn't such a big deal, but it does severely reduce go's usefulness in such environments, which seem to becoming more common.

You can the change the VM reservation amount if you really want it.

You need to modify file src/pkg/runtime/malloc.goc:

reduce macro MaxArena32 and "arena_size = 512<<20" at about line 331 in function runtime·mallocinit

Total VM reservation (in Bytes) = MaxArena32 / 8 + arena_size

Adjust these two parameters to suit your application, remember to rebuild your application after runtime changes.

(preferably with 'go install -a -v std' first)

[Richard Warburton]

Thanks minux for pointing to malloc.goc,

I copied malloc.goc to malloc.goc.backup, then made changes to malloc.goc.

Nothing seems to have made any difference...including

#define MaxArena32 (2U<<10) // from (2U<<20)

arena_size = 512<<10; // from 512<<20;

Since my VPS is 64bit, I also set:

arena_size = 16LL<<10; // from 16LL<<30;

For each change, I would:

go install -a -v std

go build sipCloak4.go

Then upload to my VPS for testing.

The changes must be going in as the md5sum of the built apps change, but it doesn't appear to be doing anything useful.

No luck so far, app still using 60+MB ram, but hope to have a break through soon!

[Dave Cheney]

> Since my VPS is 64bit, I also set:
> arena_size = 16LL<<10; // from 16LL<<30;

Hang on, if you're using 6g then none of the preceding applies as the
GC allocator behaves differently with the 64bit address space.

Lets start from the beginning, are you able to post some code that
demonstrates a 60mb working set ?

Cheers

Dave

[Richard Warburton]

Absolutely.  Just remember it's the high VM that's being counted on OpenVZ platforms.

Anyway here's a much shorter example of what I'm working on, with far less imports, yet uses about the same VM on my VPS (albeit 700k less RSS).

package main

import (

"log/syslog"

"net"

"os"

)

func main() {

logger, err := syslog.New(syslog.LOG_INFO, "VM Test") // LOG_NOTICE

if err != nil {

panic("Could not connect to SysLog: " + err.Error())

}

var port string = "5100"

if len(os.Args) > 1 {

port = os.Args[1]

}

sAddr, err := net.ResolveUDPAddr("udp", ":"+port)

if err != nil {

panic("Could not Resolve Listener")

}

conn, error := net.ListenUDP("udp", sAddr)

if error != nil {

logger.Err("Error listening on port: " + port + " " + error.Error())

panic("Error listening on port: " + port + " " + error.Error())

}

logger.Info("Started Successfully")

for { // Main Loop

incomingPacket := make([]byte, 8192)

n, addr, err := conn.ReadFromUDP(incomingPacket)

if err != nil {

logger.Warning("Could not read incoming packet: " + err.Error())

continue

}

logger.Info(addr.String() + ": " + string(incomingPacket[:n]))

}

}

Before I run the program test ram.

krolaw@alpha:~$ free

             total       used       free     shared    buffers     cached

Mem:        262144      68564     193580          0          0          0

-/+ buffers/cache:      68564     193580

Swap:            0          0          0

Now run the program.

krolaw@alpha:~$ ./memtest &

[1] 15684

krolaw@alpha:~$ free

             total       used       free     shared    buffers     cached

Mem:        262144     121576     140568          0          0          0

-/+ buffers/cache:     121576     140568

Swap:            0          0          0

krolaw@alpha:~$ ps au

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

krolaw   15679  0.0  0.7  18120  1992 pts/0    Ss   10:20   0:00 -bash

krolaw   15684  0.0  0.7  59248  1840 pts/0    Sl   10:20   0:00 ./memtest

krolaw   15688  0.0  0.4  15228  1180 pts/0    R+   10:20   0:00 ps au

krolaw@alpha:~$ killall memtest

krolaw@alpha:~$ free

             total       used       free     shared    buffers     cached

Mem:        262144      69344     192800          0          0          0

-/+ buffers/cache:      69344     192800

Swap:            0          0          0

So if there is a way to reduce the VM, it would be very helpful.  

Cheers.

[Richard Warburton]

Here's a much shorter example, that uses 30MB+ ram on OpenVZ platform.

package main

import "time"

func main() {

time.Sleep(60e9)

}

$ free

             total       used       free     shared    buffers     cached

Mem:        262144      68280     193864          0          0          0

-/+ buffers/cache:      68280     193864

Swap:            0          0          0

$ ./memtest2 &

[1] 15818

$ free

             total       used       free     shared    buffers     cached

Mem:        262144     103840     158304          0          0          0

-/+ buffers/cache:     103840     158304

Swap:            0          0          0

$ ps au      

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

krolaw   15814  0.0  0.7  18124  2028 pts/0    Ss   11:35   0:00 -bash

krolaw   15818  0.0  0.1  35820   496 pts/0    Sl   11:36   0:00 ./memtest2

krolaw   15821  0.0  0.4  15228  1184 pts/0    R+   11:36   0:00 ps au

$ killall memtest2

$ free

             total       used       free     shared    buffers     cached

Mem:        262144      69060     193084          0          0          0

-/+ buffers/cache:      69060     193084

Swap:            0          0          0

Thanks heaps.

[Dave Cheney]

Hello,

I have confirmed your results (OpenVZ host at BuffaloVPS)

deadwood(~) % ps au | grep a.out
dfc 5734 0.0 0.0 35820 496 pts/7 Sl+ 04:17 0:00
/tmp/go-build712852602/command-line-arguments/_obj/a.out

However calculating how much memory, that is how many pages of VSS
have actually be allocated by the OS (and therefore accounted against
your quota) is not as simple as adding up all the VSS rows.

deadwood(~) % pmap 5734
5734: /tmp/go-build712852602/command-line-arguments/_obj/a.out
0000000000400000 204K r-x--
/tmp/go-build252999911/command-line-arguments/_obj/a.out
0000000000433000 12K rw---
/tmp/go-build252999911/command-line-arguments/_obj/a.out
0000000000436000 32880K rw--- [ anon ]
000000f83fff0000 1088K rwx-- [ anon ]
00002b9dcd60f000 1536K rwx-- [ anon ]
00007fffe1589000 84K rw--- [ stack ]
00007fffe15fd000 12K r-x-- [ anon ]
ffffffffff600000 8192K ----- [ anon ]

I _think_ (need to check the source to confirm) that the 32mb anon map
is the heap, however this has just been mmap'ed by the OS, yet is
virtually unused, which is why the RSS value reported is less than a
megabyte.

You can confirm this yourself. In your example you are using a 256mb
openvz host, you can see that the go program is accounted for as 0.1%
of memory. For comparison the bash shell you're using is accounted for
as 0.7% of memory.

Cheers

Dave

[Richard Warburton]

Thanks Dave.  Ok, so we can see that the runtime is allocating much more ram than it needs.  Is there anything practical that can be done about this?

Thanks.

[Dave Cheney]

I don't believe the runtime is allocating more memory than is needed.
I think the fixation with the output of free is obscuring the real
issue.

Do you have a program that does not run in your 256mb VPS that has a
working set significantly less than 256mb?

Cheers

Dave

[Richard Warburton]

Actually, it's a 128MB VPS.  The 256 is total with burst.  So ideally you want to stay below 128MB most of the time.  This effectively means that I can't reliably run more than one small go program at a time.  I was thinking of writing a few services for my VPS in go, but at 60MB a pop (for a small go prog) that's not going to happen.  And anything I do write in go isn't going to be desirable to low end VPS users - which ruins a couple of project ideas...

Cheers.

--
Richard Warburton - MSc(Hons), PGDipSci, BE(Hons)

Prototec Limited - P.O. Box 41378, St Lukes, Auckland 1346, NZ +649-973-5099

[minux]

On Sat, Apr 14, 2012 at 12:40 PM, Richard Warburton <kro...@gmail.com> wrote:

Thanks Dave.  Ok, so we can see that the runtime is allocating much more ram than it needs.  Is there anything practical that can be done about this?

I think it might has something to do with cgo. You can try

CGO_ENABLED=0 ./make.bash

then rebuild your program.

[Richard Warburton]

No luck minux, but thanks for the idea.

krolaw@alpha:~$ pmap 17599

17599:   ./memtest2

0000000000400000    204K r-x--  /home/krolaw/memtest2

0000000000433000     12K rw---  /home/krolaw/memtest2

0000000000436000  32880K rw---    [ anon ]

000000f83fff0000   1088K rwx--    [ anon ]

00002b7a25e55000   1536K rwx--    [ anon ]

00007fff4fa28000     84K rw---    [ stack ]

00007fff4fa45000     12K r-x--    [ anon ]

ffffffffff600000   8192K -----    [ anon ]

 total            44008K

[Dave Cheney]

I think I was wrong this morning, the 32mb is the bitmap that
describes the heap, the actual heap is probably the rwx sections at
0xf83fff0000 or 0x2b7a25e55000.

[Rémy Oudompheng]

Le 14 avril 2012 09:36, Richard Warburton <ric...@prototec.co.nz> a écrit :
> No luck minux, but thanks for the idea.
>
> krolaw@alpha:~$ pmap 17599
> 17599:   ./memtest2
> 0000000000400000    204K r-x--  /home/krolaw/memtest2
> 0000000000433000     12K rw---  /home/krolaw/memtest2

> 0000000000436000  32880K rw---    [ anon ]

This one is a static data section:
% readelf -S gocode
...
[20] .bss NOBITS 0000000000812e78 00412e78
0000000000012d80 0000000000000000 WA 0 0 8
[21] .noptrbss NOBITS 0000000000825bf8 00425bf8
0000000002009288 0000000000000000 WA 0 0 8
...

0x2009288 = 32MB+epsilon is the size of the data block.
There must be a 32MB block somewhere in the standard library code.
You can show it by adding a print in the linker code. The 32MB block is the
runtime.mheap variable. The big part of this structure is:

struct MHeap
{
...
// span lookup
MSpan *map[1<<MHeapMap_Bits];
...
};

Which is an array of 4M pointers, hence the 32MB. I'm not sure how
tunable it is. Maybe the map could be dynamically mapped at some other
fixed address.

> 000000f83fff0000   1088K rwx--    [ anon ]

This is the heap. It is hardcoded that the heap starts at address
0xf840000000. You must be using 1024kB heap+64kB bitmap (which is
growing backwards, maybe, i'm not sure).

> 00002b7a25e55000   1536K rwx--    [ anon ]

I suppose this is your OS thread stack. What is your thread stack size
(as given by ulimit -s?) You can customize that size using ulimit. I
think Go needs a ridiculously small stack size except if you are using
cgo.

[Rémy Oudompheng]

Le 14 avril 2012 07:21, Richard Warburton <ric...@prototec.co.nz> a écrit :
> Actually, it's a 128MB VPS.  The 256 is total with burst.  So ideally you
> want to stay below 128MB most of the time.  This effectively means that I
> can't reliably run more than one small go program at a time.  I was thinking
> of writing a few services for my VPS in go, but at 60MB a pop (for a small
> go prog) that's not going to happen.  And anything I do write in go isn't
> going to be desirable to low end VPS users - which ruins a couple of project
> ideas...
>
> Cheers.

I think with such a small amount of memory it will be more efficient
to put all your services into the same Go program, especially since
they are statically linked.

Rémy.

[Richard Warburton]

$ ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 20

file size               (blocks, -f) unlimited

pending signals                 (-i) 16382

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) unlimited

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

Stack size is 8MB.  

$ ulimit -s 1024

$ ./memtest2 &

[1] 17896

$ free

             total       used       free     shared    buffers     cached

Mem:        262144     105044     157100          0          0          0

-/+ buffers/cache:     105044     157100

Swap:            0          0          0

$ killall memtest2

$ free

             total       used       free     shared    buffers     cached

Mem:        262144      69484     192660          0          0          0

-/+ buffers/cache:      69484     192660

Swap:            0          0          0

No discernible difference.  Experimented with ulimit -v 10000 previously, but that just ended up killing memtest immediately.

Thanks.

[minux]

You can tune the MHeapMap_Bits in malloc.h and arena_size in malloc.goc to reduce

memory usage, as long as they statisfy this:

(1UL << (12 + MHeapMap_Bits)) >= arena_size

(for example, I changed MHeapMap_Bits to 20, and arena_size to ”4LL<<30“, all tests

passed, and the size of bss sections is dropped to about 10MB for bin/go:

linux-amd64 go $ size bin/go # before

   text   data    bss    dec    hex    filename

4491897  97552 33672424 38261873 247d471  bin/go

linux-amd64 go $ size bin/go # after

   text   data    bss    dec    hex    filename

4491924  97552  8506600 13096076  c7d48c  bin/go

)

And for the record, this problem is not caused by the reserved VM space (which is 16GB

on 64-bit hosts, as they are mapped with PROT_NONE, i.e., they won't map to any physical

memory). Its real cause is the big runtime.mheap.

[Richard Warburton]

That's awesome minux!

As my VPS has a max of 256MB:

$ vi malloc.h

MHeapMap_Bits=16

$ vi malloc.goc

arena_size=1LL<<28

$ go install -a -v std

Then rebuilt my app which reduce its VPS footprint from 60MB to 20MB!!!

Previously:

MHeapMap_Bits=19

arena_size=1LL<<28

reduced the footprint to 24MB, so I imagine going lower has diminishing returns.

I can't help but wonder, if the memory usage is a bug, or whether these memory settings should be made available as flags in go build, or sometime in the future, the runtime will probably be improved to make good choices automatically.  Is it worth putting in a feature request/bug report?

Given that the apps RSS is 2.5MB, I'm thinking the total footprint should be around 5-10MB?  If so, there's probably more work to do and I would be happy to test.  

However, knocking off ~70% of the ram requirements is absolutely fantastic, so thanks everyone for your time and helpful suggestions.

Cheers.

[kortschak]

This looks sort of like the other face of issue 2142. Note Dmitry's comment in the thread that prompted that: https://groups.google.com/d/msg/golang-nuts/bzFaGbmCVMs/3aKl6-Hbg64J

[minux]

On Mon, Apr 16, 2012 at 6:37 PM, Richard Warburton <kro...@gmail.com> wrote:

I can't help but wonder, if the memory usage is a bug, or whether these memory settings should be made available as flags in go build, or sometime in the future, the runtime will probably be improved to make good choices automatically.  Is it worth putting in a feature request/bug report?

I don't think it is a bug. Or, put it another way, if it is a bug, then it's OpenVZ's memory statistics that is buggy.

Although we do *declare* that much memory in data sections, if we don't touch (read/write) the memory, the kernel

has no need to allocate the real (physical) memory for that page. (Quote: why charge you for something that you

*might* buy in the future *now*?)

[minux]

On Mon, Apr 16, 2012 at 7:48 PM, kortschak <dan.ko...@adelaide.edu.au> wrote:

This looks sort of like the other face of issue 2142. Note Dmitry's comment in the thread that prompted that: https://groups.google.com/d/msg/golang-nuts/bzFaGbmCVMs/3aKl6-Hbg64J

No, it is not issue 2142. Issue 2142 is about 16GB maximum heap memory on 64-bit OS.

The problem here is that OpenVZ considers allocated VM with protects other than PROT_NONE

as actual memory in use, this is hardly correct.

[Paul Borman]

The location of the bug depends on your goals and intended target.  Go's memory allocation could be considered a bug on an Embedded device with no swap or paging.  Go's penchants for allocating virtual memory it will never use makes it very difficult for the OS to make memory guarantees unless it has that logic in the page fault code.  If it has it there the only choice is to kill the process rather than reporting back to the process that it cannot have any more memory.  No, on these devices a programs should only request memory they are going to use and if given, the OS will be able to guarantee the memory will be there.

As has been previous discussed, Go is probably not the right language for such environments.  OpenVZ might be considered such an environment.  If that is the case, it probably is the wrong choice for most applications.

    -Paul

[Bobby Powers]

On Wed, Apr 11, 2012 at 6:45 PM, Richard Warburton <kro...@gmail.com> wrote:
> On a VPS with 768MB ram or larger, it's probably a non-issue.
>
> But you can do a lot with just 128MB VPS or 256MB VPS unless you are running
> Go or Java.  And with 128MB VPS' now as low as $12/year, almost anyone has
> the funds to experiment with running their own server - and there's heaps of
> mail (sendmail), dns (nsd), web (nginx), that will run beautifully on these
> systems in low ram.
>
> Now you can argue than OpenVZ just isn't cool, and you'd be right.  But it
> is the most common VPS option out there, and the only option at really low
> prices.

I pay ~$11 per month for a 613 MB t1.micro on ec2. I highly recommend it.

[Dan Kortschak]

That's not what I said.

I understand that 2142 is about available arena on 64-bit - it's one of
the two most significant issues in the issues list for me.

The point that I was making was that intelligently growing the heap is
similar to intelligently growing the arena - growing the available heap
space was Dmitry's suggestion that I was pointing to. Perhaps this is
not a useful comparison, but they are conceptually similar.

On Tue, 2012-04-17 at 23:10 +0800, minux wrote:
> On Mon, Apr 16, 2012 at 7:48 PM, kortschak

[Richard Warburton]

Ok, if this "issue" is not already listed, would someone who understands the problem and who is keen for it to be fixed, log it please?  I'll happily star it, but I don't really understand enough to adequately write about it, except go wastes ram in certain (increasingly popular) environments. 

Cheers.

--
Richard Warburton - MSc(Hons), PGDipSci, BE(Hons)

Prototec Limited - P.O. Box 41378, St Lukes, Auckland 1346, NZ +649-973-5099

[Saul Hazledine]

On Thursday, 12 April 2012 00:45:24 UTC+2, krolaw wrote:

But you can do a lot with just 128MB VPS or 256MB VPS unless you are running Go or Java.  And with 128MB VPS' now as low as $12/year, almost anyone has the funds to experiment with running their own server - and there's heaps of mail (sendmail), dns (nsd), web (nginx), that will run beautifully on these systems in low ram.

I found this thread fascinating and I share your excitement for low end VPS's but I would argue that it is still possible to get Xen hosting at reasonable prices:

 

http://www.lowendbox.com/tag/xen/

They're not quite as cheap as the OpenVZ options but are fairly priced and don't punish virtual memory allocation.

Saul Hazledine

[Richard Warburton]

Using the same logic, are Windows go issues not important because Darwin and Linux are available?

Seriously though, reasonable is a matter of opinion.  The cheapest OpenVZ VPS on Low End Box is $12/year, whilst the cheapest Xen is $32/year, so you get nearly 3 OpenVZ VPSes for the price of one Xen.  From what I have read, your can get more OpenVZ VPSes per node than Xen, so it's always going to be cheaper and more popular.

Cheers.

--
Richard Warburton - MSc(Hons), PGDipSci, BE(Hons) - http://richard.warburton.it/

GMT21-9 Skype/GTalk:krolaw NZ+6499735099 US+14247574049 UK+448452872990

P.O. Box 41378, St Lukes, Auckland 1346, New Zealand

[Kyle Lemons]

On Wed, Apr 18, 2012 at 2:51 AM, Richard Warburton <kro...@gmail.com> wrote:

Using the same logic, are Windows go issues not important because Darwin and Linux are available?

It's not an appropriate comparison; the issue here is, from our perspective, a bug in OpenVZ.  The whole point of virtual memory is that it's virtual.  Penalizing you for it seems about as helpful as charging you for pages that you print preview.

[Richard Warburton]

Perhaps a better analogy would be: "We won't fix or test our web pages for IE.  Regardless of it's dominance (admittedly eroding), the problems are IE's and tough luck to any user foolish enough to use it and visit our webpage."

Do we want to leave Go undesirable to the VPS world, because what OpenVZ is doing is wrong?  I agree it's very wrong, but it was a design choice (on their part) and isn't going to change.  

I apologize that I'm lashing out.  I've become very passionate about Go and I want to run it everywhere.

Cheers.

[Brad Fitzpatrick]

I'm not really following this thread closely, but wasn't this

http://code.google.com/p/go/issues/detail?id=2302

... which was fixed by http://code.google.com/p/go/source/detail?r=0d09d40b8f34 ?

[Richard Warburton]

No.  All my tests are with Go1.  However, with my limited understanding it looks like 2302 was a step in the right direction as Go probably wouldn't work at all on OpenVZ without it.  

[Dave Cheney]

Hi Richard,

I have run Go programs successfully on OpenVZ instances large and small, and most importantly in tandem with other memory hog programs (java, cough, java) for several years. Remys work on issue 2302 was much appreciated but had no impact on the success of compiling and running Go programs on OpenVZ. 

Dave

On 19/04/2012, at 8:30, Richard Warburton <ric...@prototec.co.nz> wrote:
Coated

[Richard Warburton]

Today I discovered a script that creates a fake swap on OpenVZ:

#!/bin/bash

SWAP="${1:-512}"

NEW="$[SWAP*1024]"; TEMP="${NEW//?/ }"; OLD="${TEMP:1}0"

umount /proc/meminfo 2> /dev/null

sed "/^Swap\(Total\|Free\):/s,$OLD,$NEW," /proc/meminfo > /etc/fake_meminfo

mount --bind /etc/fake_meminfo /proc/meminfo

free -m

credit: http://hostingfu.com/article/xen-or-openvz

Running it, before running my go programs, seems to make all my Go-OpenVZ memory problems disappear, checking in the control panel seems to confirms this. 

Can someone else confirm this (and explain what is really going on)?  It seems way too good to be true...speed, price and performance of OpenVZ but memory like Xen,KVM...

Cheers.

--
Richard Warburton - MSc(Hons), PGDipSci, BE(Hons)

Prototec Limited - P.O. Box 41378, St Lukes, Auckland 1346, NZ +649-973-5099

[Richard Warburton]

Nevermind,

root@alpha:/home/krolaw# ./memtest2 &

[1] 1455

root@alpha:/home/krolaw# ./memtest2 &

[2] 1457

root@alpha:/home/krolaw# ./memtest2 &

[3] 1459

root@alpha:/home/krolaw# ./memtest2 &

[4] 1461

throw: runtime: out of memory

All the script appears to do is fool the free command and the control panel.  The system itself continues, with using VM as RAM...

Whoops.

[krolaw]

For those poor souls following Go's memory usage OpenVZ based VPSes, the issue may fade away.  OpenVZ is changing the memory model:

http://wiki.openvz.org/VSwap

http://www.daftdroid.com/2011/12/vswap-the-new-fairer-ram-allocation-for-openvz/

I'm hoping VPS providers will quickly move to the new system, but I suspect the cheaper the VPS, the greater the wait...and I'm using a $1/month VPS.

In the meantime, I'm using the malloc.h and malloc.goc suggestions, which reduce the ram used by two thirds and should keep me going.

Many thanks to everyone for your suggestions and contributions.

[krolaw]

News!  I just tested my Go programs on an OpenVZ VPS with the more modern memory management (vSwap), and ram usage is fantastic.  It turned out that my $12/year VPS host came to the party and provided me with OpenVZ vSwap running on Centos.  So now my $1/month VPS is worth more to me than many other more expensive OpenVZ systems I am using.  (Private message me if you want to know who I used.)

So in a nutshell OpenVZ hosts that talk about burst ram are bad, especially in Go's case, whilst those that talk about vSwap are great.  Otherwise, stick with Xen or KVM.

Thanks again to all those who contributed to this thread.

[RoboTamer]

On Sunday, June 24, 2012 3:25:06 AM UTC-7, krolaw wrote:

News!  I just tested my Go programs on an OpenVZ VPS with the more modern memory management (vSwap), and ram usage is fantastic.  It turned out that my $12/year VPS host came to the party and provided me with OpenVZ vSwap running on Centos.  So now my $1/month VPS is worth more to me than many other more expensive OpenVZ systems I am using.  (Private message me if you want to know who I used.)

May I ask, who is your $12/year host? 

[Tech163]

http://www.lowendbox.com/blog/semoweb-12year-256mb-openvz-vps-in-orlando-fl/ perhaps?

Tech163

--
 
 

[Richard Warburton]

I have a semoweb openvz vps and they are not vSwap, which means that go apps are penalised very heavily in terms of ram.  I'm able to do more with my 64MB vSwap VPS than I can do with my 256MB semo VPS, at least in terms of running go programs.

Still that's a fantastic price.  The $12 vSwap company I was suggesting, was going great until they were bought out, attempted to move data centres were fraught with problems and then just disappeared...sigh.

Cheers.