16GB memory limit on amd64

[Jingcheng Zhang]

Hello gophers,

I'm running a server written in Go. It serves a lot of concurrent
connections, perhaps many millions per process.
Currently I don't know how to measure the accurate memory size used,
and when the process will reach the 16GB memory limit and panics.
The "top" command shows 2574M on VIRT, and 100M on RES.
Will the process panics when "VIRT" reaches 16G ?
Are there any detailed docs on Go's memory usage?

Thanks.

--
Best regards,
Jingcheng Zhang
Beijing, P.R.China

[Dave Cheney]

> I'm running a server written in Go. It serves a lot of concurrent
> connections, perhaps many millions per process.
> Currently I don't know how to measure the accurate memory size used,
> and when the process will reach the 16GB memory limit and panics.
> The "top" command shows 2574M on VIRT, and 100M on RES.
> Will the process panics when "VIRT" reaches 16G ?

Assuming you have more than 16Gb of free memory in your machine, and
at no point since the Go process has started it has been subject to
significant paging, then the value reported by top in the RES column
is a reasonable approximation for the amount of memory Go is using at
the moment.

Currently, the maximum size of the 64bit heap is 16Gb. Allocations
that cause the runtime to try to grow the heap above this size will
cause a panic. This limit can be raised by adjusting a few constants
in the runtime. Details are in
http://code.google.com/p/go/issues/detail?id=2142.

> Are there any detailed docs on Go's memory usage?

The best available docs are this mailing list and the source, I
suggest reading runtime/malloc.* and runtime/mgc0.c.

Cheers

Dave

[Jingcheng Zhang]

Hello Dave,

The physical memory size of the machine is 64GB, there is a lot of
free memory left, and the process is not swapping pages.
So the RES column should be the actual memory cost.

I'm reading runtime/malloc.goc but am not clear of the design of the
memory management for runtime.
So if there were some docs illustrating the details of the runtime
memory management it should be awesome.

Thanks a lot!

On Sat, Sep 1, 2012 at 12:06 PM, Dave Cheney <da...@cheney.net> wrote:
>
> Assuming you have more than 16Gb of free memory in your machine, and
> at no point since the Go process has started it has been subject to
> significant paging, then the value reported by top in the RES column
> is a reasonable approximation for the amount of memory Go is using at
> the moment.
>
> Currently, the maximum size of the 64bit heap is 16Gb. Allocations
> that cause the runtime to try to grow the heap above this size will
> cause a panic. This limit can be raised by adjusting a few constants
> in the runtime. Details are in
> http://code.google.com/p/go/issues/detail?id=2142.
>
>> Are there any detailed docs on Go's memory usage?
>
> The best available docs are this mailing list and the source, I
> suggest reading runtime/malloc.* and runtime/mgc0.c.
>
> Cheers
>
> Dave

[Dave Cheney]

Others will hopefully correct me, but I believe it is based on tcmalloc.

[Rob Pike]

On Fri, Aug 31, 2012 at 9:26 PM, Dave Cheney <da...@cheney.net> wrote:
> Others will hopefully correct me, but I believe it is based on tcmalloc.

It is.

-rob

[Rémy Oudompheng]

On 2012/9/1 Dave Cheney <da...@cheney.net> wrote:
> Currently, the maximum size of the 64bit heap is 16Gb. Allocations
> that cause the runtime to try to grow the heap above this size will
> cause a panic. This limit can be raised by adjusting a few constants
> in the runtime. Details are in
> http://code.google.com/p/go/issues/detail?id=2142.

An issue I have encountered is that the threashold for GC triggering
can end up being larger than the memory limit: when it is at 17GB, it
is never reached and you are in a memory leak state.

This is a problem when you have a system, that, for example, keeps
12GB of persistent data in memory, and regularly allocate small chunks
for working. It could be possible to lower GOGC but it doesn't look
like the correct solution. Would it be interesting to lower the growth
of the GC threshold when used memory approaches "the limit" ?

It could also benefit the 32-bit users.

Rémy.

[Sébastien Paolacci]

Yes it would. A 50% mem increase at 10MB is almost nothing, at 12GB it
just becomes dramatic.

Apart from that corner-case where not hitting the 16GB threshold is a
clearly identifiable target, guessing what is the comfort zone (before
damage) is not so easy.

Smoothing `gcpercent' over [0, min(arena_size,
/proc/meminfo.MemTotal)] might still provide with an additional
protection.

Sebastien

[Jingcheng Zhang]

Our servers will reach 16GB memory limit days later.
I decide to temporarily change the limit to 64GB. After some search on
the go-nuts list, I find a way:

File src/pkg/runtime/malloc.h:

#ifdef _64BIT
MHeapMap_Bits = 22,
#else
MHeapMap_Bits = 20,
#endif

Change 22 to 24, and

File src/pkg/runtime/malloc.goc:

arena_size = 16LL<<30;

Change 16 to 64.

I'm not sure whether this change could work correctly.
Any gophers can confirm this?

Thanks very much!

[minux]

On Thu, Sep 6, 2012 at 11:19 PM, Jingcheng Zhang <dio...@gmail.com> wrote:

Our servers will reach 16GB memory limit days later.
I decide to temporarily change the limit to 64GB. After some search on
the go-nuts list, I find a way:

File src/pkg/runtime/malloc.h:

#ifdef _64BIT
        MHeapMap_Bits = 22,
#else
        MHeapMap_Bits = 20,
#endif

Change 22 to 24, and

File src/pkg/runtime/malloc.goc:

                arena_size = 16LL<<30;

Change 16 to 64.

I'm not sure whether this change could work correctly.
Any gophers can confirm this?

confirmed. it's correct. 

ref: http://code.google.com/p/go/issues/detail?id=2142#c11

[Jingcheng Zhang]

Hello minux,

Thanks for your fast reply :-)

The comments in the issue also said that this change may cause GC
behaves unexpectedly, or slower, and will cause a "segv" error for
encoding/gob. Is this right? What side effects will these issues bring
to my code?

[minux]

On Thu, Sep 6, 2012 at 11:30 PM, Jingcheng Zhang <dio...@gmail.com> wrote:

The comments in the issue also said that this change may cause GC
behaves unexpectedly, or slower, and will cause a "segv" error for

GC will definitely be slower, as it must scan larger amount of memory.

I'm not aware of the other issues (maybe i don't test that long enough,

but imo this fix won't affect other parts)

you probably should test it before deploying though.

[Jingcheng Zhang]

Thanks very much! I'll try it tomorrow.

[Dan Kortschak]

We have been using a runtime with the memory limit set to 64GB since I
posted the current work around without any noticeable problems. The main
application that required this makes extensive use of gob, so I can say
that there don't appear to be problem relating to this.

Dan

[Jingcheng Zhang]

Hello Dan,

Thanks for your report!
I've rebuilt my Go distribution and my server binaries, it's running
now, I'll keep monitoring it.

[Dan Kortschak]

I'll just clarify, we use gob on small types and not in collections.

[Jingcheng Zhang]

Thanks Dan.

And Hello Rob, could you confirm the change has no side-effects on
"encoding/gob" ?

[Rob Pike]

Changing the arena size should have no effect on encoding/gob.

-rob

[Jingcheng Zhang]

Thanks very much :-)

On Fri, Sep 7, 2012 at 11:54 AM, Rob Pike <r...@golang.org> wrote:
> Changing the arena size should have no effect on encoding/gob.
>
> -rob

[⚛]

On Thursday, September 6, 2012 5:20:03 PM UTC+2, Jingcheng Zhang wrote:

Our servers will reach 16GB memory limit days later.
I decide to temporarily change the limit to 64GB.

Is there a reason why the memory usage of the program is going from zero to 16GB? From your description it seems that letting the program run for 1 month would result in 100GB memory usage.

[Jingcheng Zhang]

Our users are increasing everyday, causing a lot of memory used.
On the other hand, there may be some memory leaks or resource leaks in
our code, so increasing the limit gives us much time to tackle the
issues.