regarding pprof

[Venkata Bhagavatula]

Hi All,

In one of our production setups, we have configured prometheus HA on Virtual machines(node0, node1). I see that node0 prometheus takes around 5gb of ram and node1 takes just 1gb of ram.

user has changed min.block-duration to 30m and max block duration to 2h. I told them to not modify them as these are only for developers. Will this cause a RAM usage difference? But then both nodes should use more RAM.

I checked that both prometheus are using the same configuration file.  I see that the number of allocs  in node0 are more than that of node1 under the "/debug/pprof" page. 

I collected the heap dump from "curl http://localhost:9090/debug/pprof/heap".

tried to check the top heap allocations using  "go tool pprof heap_node0"

Following is the output:

File: prometheus

Type: inuse_space

Time: Jul 22, 2021 at 4:36am (EDT)

Entering interactive mode (type "help" for commands, "o" for options)

(pprof) top

Showing nodes accounting for 81.25MB, 68.48% of 118.65MB total

Dropped 68 nodes (cum <= 0.59MB)

Showing top 10 nodes out of 101

      flat  flat%   sum%        cum   cum%

      16MB 13.49% 13.49%       16MB 13.49%  github.com/prometheus/prometheus/tsdb/chunkenc.NewXORChunk

   12.34MB 10.40% 23.89%    13.94MB 11.75%  compress/flate.NewWriter

       8MB  6.75% 30.63%        8MB  6.75%  github.com/prometheus/prometheus/tsdb/chunkenc.(*bstream).writeByte

    7.50MB  6.32% 36.96%     7.50MB  6.32%  github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels

    7.50MB  6.32% 43.28%     7.50MB  6.32%  github.com/prometheus/prometheus/tsdb.newMemSeries

    7.40MB  6.24% 49.51%     7.40MB  6.24%  github.com/prometheus/prometheus/scrape.(*scrapeCache).trackStaleness

    6.50MB  5.48% 55.00%     6.50MB  5.48%  github.com/prometheus/prometheus/tsdb/chunkenc.(*bstream).writeBit

       6MB  5.06% 60.05%        6MB  5.06%  github.com/prometheus/prometheus/pkg/textparse.(*PromParser).Metric

    5.50MB  4.64% 64.69%        9MB  7.59%  github.com/prometheus/prometheus/tsdb.(*stripeSeries).getOrSet

    4.50MB  3.79% 68.48%    22.50MB 18.97%  github.com/prometheus/prometheus/tsdb.(*memSeries).cut

top output says that only ~120MB is in inuse bytes. What about the rest of the memory? I read in one of blog(https://source.coveo.com/2021/03/03/prometheus-memory/) that it is  cached memory allocated by mmap. Is it OK to have that much memory in cache?

Currently I am trying to plot a graph for go_memstats_heap_.*_bytes. 

Can you let me know how to debug further with respect to RAM usage difference between two nodes? In production setup there is a management application which monitors the total ram usage of the node and if it reaches a threshold then some action would be performed on that node. 

Thanks n Regards,

chalapathi

[Venkata Bhagavatula]

Forgot to mention, we are using prometheus version 2.16.0 and cannot update to the latest version. 

attaching the heapdump for both nodes.

[Stuart Clark]

On 23/07/2021 12:20, Venkata Bhagavatula wrote:

Forgot to mention, we are using prometheus version 2.16.0 and cannot update to the latest version. 

attaching the heapdump for both nodes.

On Fri, Jul 23, 2021 at 4:41 PM Venkata Bhagavatula <venkat...@gmail.com> wrote:

Hi All,

In one of our production setups, we have configured prometheus HA on Virtual machines(node0, node1). I see that node0 prometheus takes around 5gb of ram and node1 takes just 1gb of ram.

Are you performing queries on both nodes or just node0?

-- 
Stuart Clark

[Venkata Bhagavatula]

Queries are happening only on Node1. Node0 is only scrapping targets.

[Ben Kochie]

Are you using federation to replicate the data from node0 to node1?

That could be a major cause of excess memory use.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABXnQPtnoGU4b8r5VhCjetuO745Ukx%2BJw9fBurd8qxvPx_yi-Q%40mail.gmail.com.

[Venkata Bhagavatula]

Hi Ben

No there is no federation between node0 and node1. This is HA setup.

thanks n Regards,

chalapathi.

[Ben Kochie]

That's good. Some people have setup federation before mistakenly thinking it's HA.

Looking over your heap dumps, you're right, it seems like it's all cache. This is fine, there's no reason to worry about cache memory use. The kernel will reclaim cache if it's needed for other processes.

As for RSS use, Prometheus 2.16.0 binaries were built with a version of Go that has some differences with how RSS is reported.

I highly recommend upgrading to the current release. There have been a lot of memory use improvements since that release.