Why is there a huge gap between the memory profile's inuse_space in pprof and the externally obtained memory footprint?

[Zxilly Chou]

I'm working on the project https://github.com/Zxilly/go-size-analyzer, a program that can be very memory intensive, so I've written some scripts to fetch the pprof file during integration testing, and I'm using python's psutil to read the cpu and memory usage to automatically plot into the images.

Recently I was working on some optimizations so I started to analyze the pprof files collected in CI, my test program prints the heap profile file once per second, but I noticed that in all mem pprof files the process memory footprint varies dramatically for both inuse_space and psutil reads. Here are two images showing the data read by the same test.

As shown in the figure, the difference in size between the memory read by psutil and the memory fetched by pprof is almost 800mb. I know that mem pprof can't count data on the stack, but I don't think the stack holds that much.

You can download the data collected by CI and the rendered images at https://github.com/Zxilly/go-size-analyzer/actions/runs/10007859171/artifacts/1718981276. The data is in the cockroach-linux-amd64/json directory in the tarball.

Below is the key code for collecting data.

python part:

    ps_process = psutil.Process(process.pid)
   
    while process.poll() is None:
        percent = ps_process.cpu_percent(interval=0.1)
        mem = ps_process.memory_info().rss / (1024 * 1024)
        elapsed_time = time.time() - start_time
        if elapsed_time > timeout:
            raise TimeoutError(f"Process {name} timed out after {timeout} seconds.")
   
        if draw:
            cpu_percentages.append(percent)
            memory_usage_mb.append(mem)
            timestamps.append(elapsed_time)

golang part:

    var ctx context.Context
    ctx, heapProfileStop = context.WithCancel(context.Background())
   
    go func() {
        ticker := time.NewTicker(1 * time.Second)
        defer ticker.Stop()
   
        id := 0
        write := func() {
           id++
           path := filepath.Join(outputDir, fmt.Sprintf("mem-%d.pprof", id))
           f, err := os.Create(path)
           defer func(f *os.File) {
              err = f.Close()
              if err != nil {
                 panic(err)
              }
           }(f)
           if err != nil {
              panic(err)
           }
   
           err = pprof.WriteHeapProfile(f)
           if err != nil {
              panic(err)
           }
        }
   
        for {
           select {
           case <-ctx.Done():
              write()
              return
           case <-ticker.C:
              write()
           }
        }
    }()