Should the "Sys" stat or any other stat/combination reported by runtime.ReadMemStatsapproximately equal the resident memory set reported by ps aux?

[evan....@go2mobi.com]

(cross posted from http://stackoverflow.com/questions/37352764/should-the-stats-reported-by-gos-runtime-readmemstats-approximately-equal-the-r)

Should the "Sys" stat or any other stat/combination reported by runtime.ReadMemStatsapproximately equal the resident memory set reported by ps aux?

Alternatively, assuming some memory may be swapped out, should the Sys stat be approximately greater than or equal to the RSS?

We have a long-running web service that deals with a high frequency of requests and we are finding that the RSS quickly climbs up to consume virtually all of the 64GB memory on our servers. When it hits ~85% we begin to experience considerable degradation in our response times and in how many concurrent requests we can handle. The run I've listed below is after about 20 hours of execution, and is already at 51% memory usage.

I'm trying to determine if the likely cause is a memory leak (we make some calls to CGO). The data seems to indicate that it is, but before I go down that rabbit hole I want to rule out a fundamental misunderstanding of the statistics I'm using to make that call.

This is an amd64 build targeting linux and executing on CentOS.

Reported by runtime.ReadMemStats:

Alloc:     1294777080 bytes (1234.80MB)  // bytes allocated and not yet freed
Sys:       3686471104 bytes (3515.69MB)  // bytes obtained from system (sum of XxxSys below)
HeapAlloc: 1294777080 bytes (1234.80MB)  // bytes allocated and not yet freed (same as Alloc above)
HeapSys:   3104931840 bytes (2961.09MB)  // bytes obtained from system
HeapIdle:  1672339456 bytes (1594.87MB)  // bytes in idle spans
HeapInuse: 1432592384 bytes (1366.23MB)  // bytes in non-idle span

Reported by ps aux:

%CPU %MEM       VSZ      RSS 
1362 51.3 306936436 33742120

[Dave Cheney]

The values reported by runtime.memstats are accurate. The values from top are often innacurate.

[evan....@go2mobi.com]

Would the system be off by 10x-20x? After another 24 hours the system is reporting that the memory is almost completely used (55gb of 64gb) by the go process, but the go memstats are reporting only 4gb used max by all measurements. 

The heap shows what I'd expect to see on the heap, and the memstats closely mirror the heap report from pprof. 

My assumption is still memory leak, but I've eliminated the CGO code (100% go code now) and I'm still seeing the problem.

So strange!

My original question was simply whether or not I should trust my gut that it's a memory leak, but if it's 100% Go code now where else should I be looking based on this data?

Are there any core Go ways to leak memory that wouldn't be reported by memstats? Syscalls maybe?

I will of course be auditing all 3rd party libs as well as our own code at this point

Thank you!

[Dave Cheney]

If runtime.memstats is not growing, and you are not leaking operating system threads then it's top that is reporting the incorrect value.

[Jakob Borg]

2016-05-22 19:37 GMT+09:00 Dave Cheney <da...@cheney.net>:
> If runtime.memstats is not growing, and you are not leaking operating system threads then it's top that is reporting the incorrect value.

Or, as you already concluded, you're leaking memory in the C side of things. :)

//jb

[James Bardin]

On Saturday, May 21, 2016 at 11:42:00 PM UTC-4, evan....@go2mobi.com wrote:

My original question was simply whether or not I should trust my gut that it's a memory leak, but if it's 100% Go code now where else should I be looking based on this data?

Are there any core Go ways to leak memory that wouldn't be reported by memstats? Syscalls maybe?

Are you certain it's 100% Go now? Have you compiled everything with CGO_ENABLED=0 (you can use -installsuffix to help ensure you're getting a clean install). If there's still large chunks of unaccounted for memory, looking for packages that call syscall.Mmap would be my next step.

[evan....@go2mobi.com]

Are you certain it's 100% Go now? Have you compiled everything with CGO_ENABLED=0 (you can use -installsuffix to help ensure you're getting a clean install). If there's still large chunks of unaccounted for memory, looking for packages that call syscall.Mmap would be my next step.

Apparently I'm awful at using google groups. Third time is a charm. Apologies for the post-delete-post-delete. Posted from personal, second had my email signature unnecessarily. Here is the post:

I'll try disabling CGO to make sure it's not in use--I had only commented out CGO calls.

We do have a memory mapped file, but I have already eliminated that as a possibility by disabling it.

I'll check out our dependencies for mmap as well, but I'll be shocked if they are mapping 50gb+ files.

Thanks for the suggestions so far!

[Nick Craig-Wood]

On 20/05/16 19:24, evan....@go2mobi.com wrote:
> Reported by |ps aux|:
>
> %CPU %MEM VSZ RSS

> 1362 51.3 306,936,436 33,742,120

33GB RSS, but 306GB VSZ? Why is your program using 10x the virtual
memory than resident memory? That looks suspicious.

Is it memory mapping a very big (say 250GB) file maybe?

If you use that file you'd expect it to be swapped into memory when it
is used and memory pressure on the OS to swap out pages as necessary.

--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick