Re: GO program's memory footprint is confusing

[garenchan]

Both hosts have 8 cores and 16GB RAM.

在2022年4月30日星期六 UTC+8 00:19:44<garenchan> 写道：

What version of Go are you using (go version)?

$ go version
go version go1.17.6 linux/amd64

Does this issue reproduce with the latest release?

uncertain

What operating system and processor architecture are you using (go env)?

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY=""
GOROOT="/home/go"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/home/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/demo/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build4023324410=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I encountered a memory problem with the GO program, see here for details.(https://stackoverflow.com/questions/71994609/memory-footprint-of-the-same-program-varies-greatly-in-two-similar-environments)

In order to simplify the analysis, I wrote a simple program to test.

```go
package main

import (
    "time"
)

func main() {
    time.Sleep(60*time.Second)
}
```

• I compiled it into binary file on a linux host `host1` with kernel 4.18. Then I run it on `host1` and the process takes up close to 5MB RSS.
• I then copy the binary file to another host `host2` with kernel 4.18. I also ran it on `host2`, but this time the process took up less than 1MB RSS.
• I repeated the test many times and observed the same thing.

```
$ uname -a
Linux host1 4.18.0 #1 SMP Wed Nov 10 20:46:19 CST 2021 x86_64 x86_64 x86_64 GNU/Linux

$ uname -a
Linux host2 4.18.0 #1 SMP Fri May 8 10:59:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
```

Why is memory footprint of the same program in similar environments so different? What factors might be contributing to this problem?

What did you expect to see?

I would expect to see the memory footprint of the same program in similar environments be close. I look forward to your answers. Thank you very much.

[jake...@gmail.com]

Since no one has responded with concrete ideas, I'll throw out two suggestions. They may seem obvious.

 First have you tries the latest version of Go? and do you get the same results?

 Second have you run the experiment with a small binaries not from Go? I would suggest something that does allocate some real memory, not a "hello world" C program or something.

[garenchan]

@jake, I tried 1.18 version of Go, but the problem still exists. 
I also tried using C to write program that includes memory allocation and usage,  and this problem does not occur.

[Robert Engels]

Are you certain that on the low memory instance there isn’t another process - maybe Go - that hasn’t already loaded the shared libraries - and on the other it hasn’t - so the RSS is different. Compare the VSZ sizes and see if they are the same. 

On May 5, 2022, at 10:28 PM, garenchan <garen...@gmail.com> wrote:

@jake, I tried 1.18 version of Go, but the problem still exists. 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d8a3e1d9-e578-4668-9b19-38b49e29cd11n%40googlegroups.com.

[garenchan]

The binary file is not a dynamic executable.   So I don't think that will happen.

[Robert Engels]

https://www.arp242.net/static-go.html

On May 5, 2022, at 11:27 PM, garenchan <garen...@gmail.com> wrote:

The binary file is not a dynamic executable.   So I don't think that will happen.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/c524a157-7772-4bcd-9ce5-c04b95fb560en%40googlegroups.com.

[Uli Kunitz]

I cannot say what the difference is, but I would do two things to solve the mystery:

1) Compare /proc/<pid>/limits for differences.

2) Compare the output of egrep '^Rss:|^[0-9a-f]' /proc/<pid>/smaps for the processes on the different machines.

Report the results and share the outputs.

[garenchan]

/proc/<pid>/limits  of the two processes are the same.

But /proc/<pid>/smaps of the two processes are quite different, I have uploaded them to the attachment.

[Uli Kunitz]

I didn't found the attachment here. Can you put it somewhere. Maybe as github gist?

[David Finkel]

Looking at proc-10451, I see that there are two mappings with anonymous huge pages in that smaps output, and none on the other.
Does one machine have transparent hugepages enabled, and the other not?

You can check whether they're enabled on that system by looking at /sys/kernel/mm/transparent_hugepage/enabled.

Here are the relevant smaps sections:

c000000000-c000400000 rw-p 00000000 00:00 0
Size:               4096 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                2048 kB
Pss:                2048 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:      2048 kB
Referenced:         2048 kB
Anonymous:          2048 kB
LazyFree:              0 kB
AnonHugePages:      2048 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac sd

7f716a5e2000-7f716c953000 rw-p 00000000 00:00 0
Size:              36292 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Rss:                2092 kB
Pss:                2092 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:      2092 kB
Referenced:         2092 kB
Anonymous:          2092 kB
LazyFree:              0 kB
AnonHugePages:      2048 kB
ShmemPmdMapped:        0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac sd

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/daba9d60-7c45-45f3-83c8-afc2471841c6n%40googlegroups.com.