I'm running a server written in Go. It serves a lot of concurrent
connections, perhaps many millions per process.
Currently I don't know how to measure the accurate memory size used,
and when the process will reach the 16GB memory limit and panics.
The "top" command shows 2574M on VIRT, and 100M on RES.
Will the process panics when "VIRT" reaches 16G ?
Are there any detailed docs on Go's memory usage?
Thanks.
-- Best regards,
Jingcheng Zhang
Beijing, P.R.China
> I'm running a server written in Go. It serves a lot of concurrent
> connections, perhaps many millions per process.
> Currently I don't know how to measure the accurate memory size used,
> and when the process will reach the 16GB memory limit and panics.
> The "top" command shows 2574M on VIRT, and 100M on RES.
> Will the process panics when "VIRT" reaches 16G ?
Assuming you have more than 16Gb of free memory in your machine, and
at no point since the Go process has started it has been subject to
significant paging, then the value reported by top in the RES column
is a reasonable approximation for the amount of memory Go is using at
the moment.
Currently, the maximum size of the 64bit heap is 16Gb. Allocations
that cause the runtime to try to grow the heap above this size will
cause a panic. This limit can be raised by adjusting a few constants
in the runtime. Details are in
http://code.google.com/p/go/issues/detail?id=2142.
> Are there any detailed docs on Go's memory usage?
The best available docs are this mailing list and the source, I
suggest reading runtime/malloc.* and runtime/mgc0.c.
The physical memory size of the machine is 64GB, there is a lot of
free memory left, and the process is not swapping pages.
So the RES column should be the actual memory cost.
I'm reading runtime/malloc.goc but am not clear of the design of the
memory management for runtime.
So if there were some docs illustrating the details of the runtime
memory management it should be awesome.
On Sat, Sep 1, 2012 at 12:06 PM, Dave Cheney <d...@cheney.net> wrote:
> Assuming you have more than 16Gb of free memory in your machine, and
> at no point since the Go process has started it has been subject to
> significant paging, then the value reported by top in the RES column
> is a reasonable approximation for the amount of memory Go is using at
> the moment.
> Currently, the maximum size of the 64bit heap is 16Gb. Allocations
> that cause the runtime to try to grow the heap above this size will
> cause a panic. This limit can be raised by adjusting a few constants
> in the runtime. Details are in
> http://code.google.com/p/go/issues/detail?id=2142.
>> Are there any detailed docs on Go's memory usage?
> The best available docs are this mailing list and the source, I
> suggest reading runtime/malloc.* and runtime/mgc0.c.
> Cheers
> Dave
-- Best regards,
Jingcheng Zhang
Beijing, P.R.China
> The physical memory size of the machine is 64GB, there is a lot of
> free memory left, and the process is not swapping pages.
> So the RES column should be the actual memory cost.
> I'm reading runtime/malloc.goc but am not clear of the design of the
> memory management for runtime.
> So if there were some docs illustrating the details of the runtime
> memory management it should be awesome.
> Thanks a lot!
> On Sat, Sep 1, 2012 at 12:06 PM, Dave Cheney <d...@cheney.net> wrote:
>> Assuming you have more than 16Gb of free memory in your machine, and
>> at no point since the Go process has started it has been subject to
>> significant paging, then the value reported by top in the RES column
>> is a reasonable approximation for the amount of memory Go is using at
>> the moment.
>> Currently, the maximum size of the 64bit heap is 16Gb. Allocations
>> that cause the runtime to try to grow the heap above this size will
>> cause a panic. This limit can be raised by adjusting a few constants
>> in the runtime. Details are in
>> http://code.google.com/p/go/issues/detail?id=2142.
>>> Are there any detailed docs on Go's memory usage?
>> The best available docs are this mailing list and the source, I
>> suggest reading runtime/malloc.* and runtime/mgc0.c.
> Currently, the maximum size of the 64bit heap is 16Gb. Allocations
> that cause the runtime to try to grow the heap above this size will
> cause a panic. This limit can be raised by adjusting a few constants
> in the runtime. Details are in
> http://code.google.com/p/go/issues/detail?id=2142.
An issue I have encountered is that the threashold for GC triggering
can end up being larger than the memory limit: when it is at 17GB, it
is never reached and you are in a memory leak state.
This is a problem when you have a system, that, for example, keeps
12GB of persistent data in memory, and regularly allocate small chunks
for working. It could be possible to lower GOGC but it doesn't look
like the correct solution. Would it be interesting to lower the growth
of the GC threshold when used memory approaches "the limit" ?
Yes it would. A 50% mem increase at 10MB is almost nothing, at 12GB it
just becomes dramatic.
Apart from that corner-case where not hitting the 16GB threshold is a
clearly identifiable target, guessing what is the comfort zone (before
damage) is not so easy.
Smoothing `gcpercent' over [0, min(arena_size,
/proc/meminfo.MemTotal)] might still provide with an additional
protection.
<remyoudomph...@gmail.com> wrote:
> On 2012/9/1 Dave Cheney <d...@cheney.net> wrote:
>> Currently, the maximum size of the 64bit heap is 16Gb. Allocations
>> that cause the runtime to try to grow the heap above this size will
>> cause a panic. This limit can be raised by adjusting a few constants
>> in the runtime. Details are in
>> http://code.google.com/p/go/issues/detail?id=2142.
> An issue I have encountered is that the threashold for GC triggering
> can end up being larger than the memory limit: when it is at 17GB, it
> is never reached and you are in a memory leak state.
> This is a problem when you have a system, that, for example, keeps
> 12GB of persistent data in memory, and regularly allocate small chunks
> for working. It could be possible to lower GOGC but it doesn't look
> like the correct solution. Would it be interesting to lower the growth
> of the GC threshold when used memory approaches "the limit" ?
Our servers will reach 16GB memory limit days later.
I decide to temporarily change the limit to 64GB. After some search on
the go-nuts list, I find a way:
On Sat, Sep 1, 2012 at 12:06 PM, Dave Cheney <d...@cheney.net> wrote:
>> I'm running a server written in Go. It serves a lot of concurrent
>> connections, perhaps many millions per process.
>> Currently I don't know how to measure the accurate memory size used,
>> and when the process will reach the 16GB memory limit and panics.
>> The "top" command shows 2574M on VIRT, and 100M on RES.
>> Will the process panics when "VIRT" reaches 16G ?
> Assuming you have more than 16Gb of free memory in your machine, and
> at no point since the Go process has started it has been subject to
> significant paging, then the value reported by top in the RES column
> is a reasonable approximation for the amount of memory Go is using at
> the moment.
> Currently, the maximum size of the 64bit heap is 16Gb. Allocations
> that cause the runtime to try to grow the heap above this size will
> cause a panic. This limit can be raised by adjusting a few constants
> in the runtime. Details are in
> http://code.google.com/p/go/issues/detail?id=2142.
>> Are there any detailed docs on Go's memory usage?
> The best available docs are this mailing list and the source, I
> suggest reading runtime/malloc.* and runtime/mgc0.c.
> Cheers
> Dave
-- Best regards,
Jingcheng Zhang
Beijing, P.R.China
On Thu, Sep 6, 2012 at 11:19 PM, Jingcheng Zhang <dio...@gmail.com> wrote:
> Our servers will reach 16GB memory limit days later.
> I decide to temporarily change the limit to 64GB. After some search on
> the go-nuts list, I find a way:
The comments in the issue also said that this change may cause GC
behaves unexpectedly, or slower, and will cause a "segv" error for
encoding/gob. Is this right? What side effects will these issues bring
to my code?
On Thu, Sep 6, 2012 at 11:25 PM, minux <minux...@gmail.com> wrote:
> On Thu, Sep 6, 2012 at 11:19 PM, Jingcheng Zhang <dio...@gmail.com> wrote:
>> Our servers will reach 16GB memory limit days later.
>> I decide to temporarily change the limit to 64GB. After some search on
>> the go-nuts list, I find a way:
On Thu, Sep 6, 2012 at 11:30 PM, Jingcheng Zhang <dio...@gmail.com> wrote:
> The comments in the issue also said that this change may cause GC
> behaves unexpectedly, or slower, and will cause a "segv" error for
GC will definitely be slower, as it must scan larger amount of memory.
I'm not aware of the other issues (maybe i don't test that long enough,
but imo this fix won't affect other parts)
you probably should test it before deploying though.
On Thu, Sep 6, 2012 at 11:56 PM, minux <minux...@gmail.com> wrote:
> On Thu, Sep 6, 2012 at 11:30 PM, Jingcheng Zhang <dio...@gmail.com> wrote:
>> The comments in the issue also said that this change may cause GC
>> behaves unexpectedly, or slower, and will cause a "segv" error for
> GC will definitely be slower, as it must scan larger amount of memory.
> I'm not aware of the other issues (maybe i don't test that long enough,
> but imo this fix won't affect other parts)
> you probably should test it before deploying though.
>> encoding/gob. Is this right? What side effects will these issues bring
>> to my code?
-- Best regards,
Jingcheng Zhang
Beijing, P.R.China
We have been using a runtime with the memory limit set to 64GB since I
posted the current work around without any noticeable problems. The main
application that required this makes extensive use of gob, so I can say
that there don't appear to be problem relating to this.
On Thu, 2012-09-06 at 23:56 +0800, minux wrote:
> On Thu, Sep 6, 2012 at 11:30 PM, Jingcheng Zhang <dio...@gmail.com> wrote:
> > The comments in the issue also said that this change may cause GC
> > behaves unexpectedly, or slower, and will cause a "segv" error for
> GC will definitely be slower, as it must scan larger amount of memory.
> I'm not aware of the other issues (maybe i don't test that long enough,
> but imo this fix won't affect other parts)
> you probably should test it before deploying though.
> > encoding/gob. Is this right? What side effects will these issues bring
> > to my code?
<dan.kortsc...@adelaide.edu.au> wrote:
> We have been using a runtime with the memory limit set to 64GB since I
> posted the current work around without any noticeable problems. The main
> application that required this makes extensive use of gob, so I can say
> that there don't appear to be problem relating to this.
> Dan
> On Thu, 2012-09-06 at 23:56 +0800, minux wrote:
>> On Thu, Sep 6, 2012 at 11:30 PM, Jingcheng Zhang <dio...@gmail.com> wrote:
>> > The comments in the issue also said that this change may cause GC
>> > behaves unexpectedly, or slower, and will cause a "segv" error for
>> GC will definitely be slower, as it must scan larger amount of memory.
>> I'm not aware of the other issues (maybe i don't test that long enough,
>> but imo this fix won't affect other parts)
>> you probably should test it before deploying though.
>> > encoding/gob. Is this right? What side effects will these issues bring
>> > to my code?
-- Best regards,
Jingcheng Zhang
Beijing, P.R.China
On Thursday, September 6, 2012 5:20:03 PM UTC+2, Jingcheng Zhang wrote: > Our servers will reach 16GB memory limit days later. > I decide to temporarily change the limit to 64GB.
Is there a reason why the memory usage of the program is going from zero to 16GB? From your description it seems that letting the program run for 1 month would result in 100GB memory usage.
Our users are increasing everyday, causing a lot of memory used.
On the other hand, there may be some memory leaks or resource leaks in
our code, so increasing the limit gives us much time to tackle the
issues.
On Fri, Sep 7, 2012 at 7:36 PM, ⚛ <0xe2.0x9a.0...@gmail.com> wrote:
> On Thursday, September 6, 2012 5:20:03 PM UTC+2, Jingcheng Zhang wrote:
>> Our servers will reach 16GB memory limit days later.
>> I decide to temporarily change the limit to 64GB.
> Is there a reason why the memory usage of the program is going from zero to
> 16GB? From your description it seems that letting the program run for 1
> month would result in 100GB memory usage.
-- Best regards,
Jingcheng Zhang
Beijing, P.R.China
