Re: [go-nuts] memstats.HeapInUse is different from total in pprof

[Ian Lance Taylor]

On Mon, Sep 2, 2019 at 11:01 PM Chao Yuepan <smal...@gmail.com> wrote:
>
> I have run a programas redis proxy which parses redis commands and redirects to redis servers.
>
> I noticed it looks has memory leak issue.
>
> I use pprof to check the memory usage. It sows the total memory usage is 856.02MB. I understand the value is memory usage in heap.
>
>
>
>
>
> If I check the runtime.memstats, it show the heapInUse is about 1.7GB.
>
>
>
> I use top to check RES of this program, it is 1.7G , it isthe same to memstats.HeapInuse.
>
>
> I'm surprised why the total value in pprof is very different from the real memory usage? (856.02MB vs 1.7G)
>
> And How can I figure out those usage in heap which are not displayed in pprof?

HeapInUse tells you the amount of system memory allocated to hold Go
objects, which includes space allocated to hold objects that do not
yet exist or that have been released by the garbage collector. pprof
tells you about live memory that is accessible by your program. With
the default GOGC value of 100, in a program in steady state, it's
normal for HeapInUse to be around twice as much as pprof memory, and
that is what you are seeing.

Ian

[smallnest]

Got it,thanks

发自我的小米手机

在 Ian Lance Taylor <ia...@golang.org>，2019年9月3日 21:28写道：

[varun...@gmail.com]

@Ian,

This statement is a bit confusing to me regarding HeapInuse: "which includes space allocated to hold objects that do not yet exist or that have been released by the garbage collector"

According to memstats documentation, HeapInUse -> Includes all spans that have at least one object in them. 

So, why should it include space for objects that do not exist any more or that has been released by garbage collector. Should it not go to HeapIdle?

(Or)

Is it the case of internal fragmentation. For example, if a span is shared by 2 objects and one object is freed - then, the entire span is accounted under HeapInUse. But since there is free space in span, it can be used by new objects and it can not be returned to HeapIdle as there is one object in the span. When pprof runs, it only uses the live objects and therefore, it does not account for the free space in spans while HeapInUse will account for it. 

If this is the case, then how can we know the percentage of utilisation across a size class. 

Kindly clarify.

Thanks,

Varun

[Ian Lance Taylor]

On Tue, May 25, 2021 at 6:53 AM varun...@gmail.com <varun...@gmail.com> wrote:
>
> This statement is a bit confusing to me regarding HeapInuse: "which includes space allocated to hold objects that do not yet exist or that have been released by the garbage collector"
>
> According to memstats documentation, HeapInUse -> Includes all spans that have at least one object in them.
>
> So, why should it include space for objects that do not exist any more or that has been released by garbage collector. Should it not go to HeapIdle?

A typical span can contain a large number of objects. Each one of
those objects may be allocated or not. If a span contains at least
one allocated object, the total size of the span will be counted as
part of HeapInUse.

> Is it the case of internal fragmentation. For example, if a span is shared by 2 objects and one object is freed - then, the entire span is accounted under HeapInUse. But since there is free space in span, it can be used by new objects and it can not be returned to HeapIdle as there is one object in the span. When pprof runs, it only uses the live objects and therefore, it does not account for the free space in spans while HeapInUse will account for it.

Yes.

> If this is the case, then how can we know the percentage of utilisation across a size class.

That seems like a different question. But you can find the answer to
that question by looking at the BySize array.

Ian

[varun...@gmail.com]

Thanks @Ian for the clarification. 

The BySize arrays gives the number of active objects in a size class but it does not give the total memory allocated for a size class. E.g, for 1k size class, if 100K was the total size allocated and there are only 10 objects in it, it does not tell the memory consumption of these 10 objects.

I partially got what I am looking from the core dump. Taking core dump of the process and analysing it with viewcore shows the follows:

(viewcore) breakdown

 all                  11055939584 100.00%

   text                  28172288   0.25%

   readonly            7422455808  67.14%

   data                   3137536   0.03%

   bss                 1497206784  13.54% (grab bag, includes OS thread stacks, ...)

   heap                2037923840  18.43%

     in use spans      1340497920  12.12%

       alloc           1306139856  11.81%

         live           269335312   2.44%

         garbage       1036804544   9.38%

       free              19920896   0.18%

       round             14437168   0.13%

     manual spans       424214528   3.84% (Go stacks)

       alloc            420808704   3.81%

       free               3405824   0.03%

     free spans         273211392   2.47%

       retained         256065536   2.32% (kept for reuse by Go)

       released          17145856   0.16% (given back to the OS)

   ptr bitmap            65011712   0.59%

   span table             2031616   0.02%

(viewcore)

Cross referencing the output with this comment: https://github.com/golang/go/issues/32284#issuecomment-497851388, alloc -> heap -> in use spans seems to be HeapInUse. In the alloc of 1306139856 bytes, there seems to be 1036804544 garbage. 

I have a follow-up question on this:

For the memory profile taken around the same time as core dump, the total memory seems to be 688M

(pprof) top 10
Showing nodes accounting for 592.62MB, 86.06% of 688.58MB total
Dropped 178 nodes (cum <= 3.44MB)

alloc -> heap -> in use spans -> live only accounts for 269M. Is the rest ~420M in the profile coming from StackInUse (alloc -> heap -> manual spans)?

Is it always the case that StackInUse is shown in heap profile ?

Thanks,

Varun

[varun...@gmail.com]

(Sorry for the typo. Where ever I mentioned alloc -> heap, I mean all -> heap)

[Ian Lance Taylor]

I'm not sure, sorry.

Ian