k8s service discovery and memory usage

[Kevin Black]

I have some namespaces with just a few thousand metrics, but I have a couple gigs higher memory usage over comparable prometheus instances.  The only difference I can find is that the instance with high memory usage is in a k8s environment with significantly more pods/services (~57,000).  We use k8s pod/service discovery, but can't filter to namespaces since the targets will live in different namespaces in a dynamic fashion.  So we rely on pod/service discovery, and then use relabel_configs with a keep action on a specific label which keeps a handful of targets.  Is it known/expected that a large discovery even without actually scraping the metrics would have a couple GB memory impact?  Is there something else we could be doing to mitigate this usage?

[Simon Pasquier]

I don't recall that anybody mentioned something similar but taking a
memory profile of the running server would useful.
You can use promtool for this:
promtool debug pprof <Prometheus URL>

On Sun, Sep 15, 2019 at 10:51 PM Kevin Black <kevinbla...@gmail.com> wrote:
>
> I have some namespaces with just a few thousand metrics, but I have a couple gigs higher memory usage over comparable prometheus instances. The only difference I can find is that the instance with high memory usage is in a k8s environment with significantly more pods/services (~57,000). We use k8s pod/service discovery, but can't filter to namespaces since the targets will live in different namespaces in a dynamic fashion. So we rely on pod/service discovery, and then use relabel_configs with a keep action on a specific label which keeps a handful of targets. Is it known/expected that a large discovery even without actually scraping the metrics would have a couple GB memory impact? Is there something else we could be doing to mitigate this usage?
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f66c1eb9-2773-45ba-81d4-2a7e2c3fc6c3%40googlegroups.com.

[Kevin Black]

Thanks for the pointer.  I'm not familiar with the functions, but as I look through some of them, they do seem like they come into play with discovery.  There are lightly used pods, almost no query traffic, only ~5k metrics with 3.5GB Ram.  The dump shows way less ram used, but these still end up OOMing fairly frequently. From the heap, it looks like these are the top for memory usage:

  255.98MB 30.88% 30.88%   255.98MB 30.88%  github.com/prometheus/prometheus/pkg/labels.(*Builder).Labels
  111.74MB 13.48% 44.36%   111.74MB 13.48%  reflect.unsafe_NewArray
   90.50MB 10.92% 55.28%    98.02MB 11.83%  github.com/json-iterator/go.(*Iterator).ReadString
   74.74MB  9.02% 64.30%    74.74MB  9.02%  github.com/prometheus/common/model.LabelSet.Merge
   64.04MB  7.73% 72.02%    64.04MB  7.73%  reflect.mapassign
   50.53MB  6.10% 78.12%    51.03MB  6.16%  github.com/prometheus/prometheus/discovery/kubernetes.podLabels
   31.07MB  3.75% 81.87%    31.07MB  3.75%  github.com/prometheus/prometheus/pkg/labels.New
   30.56MB  3.69% 85.55%   332.62MB 40.13%  github.com/prometheus/prometheus/scrape.targetsFromGroup
   25.06MB  3.02% 88.58%    25.06MB  3.02%  github.com/prometheus/prometheus/scrape.(*Target).DiscoveredLabels
      18MB  2.17% 90.75%       18MB  2.17%  github.com/modern-go/reflect2.(*unsafeType).UnsafeNew
       9MB  1.09% 91.83%        9MB  1.09%  github.com/prometheus/prometheus/scrape.NewTarget
       8MB  0.97% 92.80%        8MB  0.97%  reflect.makemap
    7.52MB  0.91% 93.71%     7.52MB  0.91%  github.com/json-iterator/go.(*Iterator).readStringSlowPath
    5.50MB  0.66% 94.37%    32.03MB  3.86%  github.com/prometheus/prometheus/discovery/kubernetes.(*Pod).buildPod
    5.50MB  0.66% 95.03%     5.50MB  0.66%  github.com/prometheus/prometheus/pkg/labels.NewBuilder
    0.58MB  0.07% 95.10%   358.25MB 43.22%  github.com/prometheus/prometheus/scrape.(*scrapePool).Sync
    0.50MB  0.06% 95.16%    36.59MB  4.41%  github.com/prometheus/prometheus/discovery/kubernetes.(*Endpoints).buildEndpoints.func1
    0.50MB  0.06% 95.22%   108.74MB 13.12%  github.com/prometheus/prometheus/discovery/kubernetes.(*Endpoints).process

    0.50MB  0.06% 95.28%    32.53MB  3.92%  github.com/prometheus/prometheus/discovery/kubernetes.(*Pod).process

I guess we'll need to look at different ways of filtering here that might make more sense, since discovering almost 60k hosts/services just to keep/scan just a handful for these hundred small instances seems to be a bad idea.

On Monday, September 16, 2019 at 3:35:39 AM UTC-4, Simon Pasquier wrote:

I don't recall that anybody mentioned something similar but taking a
memory profile of the running server would useful.
You can use promtool for this:
promtool debug pprof <Prometheus URL>

On Sun, Sep 15, 2019 at 10:51 PM Kevin Black <kevinbla...@gmail.com> wrote:
>
> I have some namespaces with just a few thousand metrics, but I have a couple gigs higher memory usage over comparable prometheus instances.  The only difference I can find is that the instance with high memory usage is in a k8s environment with significantly more pods/services (~57,000).  We use k8s pod/service discovery, but can't filter to namespaces since the targets will live in different namespaces in a dynamic fashion.  So we rely on pod/service discovery, and then use relabel_configs with a keep action on a specific label which keeps a handful of targets.  Is it known/expected that a large discovery even without actually scraping the metrics would have a couple GB memory impact?  Is there something else we could be doing to mitigate this usage?
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Kevin Black]

I found a namespace with a small prometheus instance that currently had no targets at all to scrape currently, so would discover ~60k targets, but then drop them all before actually scraping them.  I verified it was getting no metrics from the jobs where the drop was happening, then removed those (unused) job configs.  I made no other changes, and my memory utilization dropped by 1.2 GB. For our larger instances this wouldn't matter, but we run a lot of small instances too, so for these we'll have to figure out something better for their target discovery.