Is there a way to limit maximum memory usage of prometheus service?

[Rnmkr]

Hello everyone, I'm new to prometheus. I like this tool.

I'm curious about how to control prometheus resource usage. Right now  I've successfully managed to limit prometheus disk usage by --storage.tsdb.retention flag.(https://prometheus.io/docs/prometheus/latest/storage/)

There is a topic on v1.8 doc (https://prometheus.io/docs/prometheus/1.8/storage/#memory-usage) but unfortunately there isn't information on 2.3 doc.

Right now Prometheus service's memory usage is increasing constantly as time goes by. My server has 2GB memory and there are other services running on this machine. If it's possible I'd like to limit prometheus memory usage. Is it possible? How prometheus manages its memory usage?

[ryans...@gmail.com]

Yes you can. Please see the below config for my prometheus deployment iin compose.yml file.

services:

  mprometheus:

    image: ${REGISTRY}/prometheus:2.1.0

    command: 

      - "--config.file=/etc/prometheus/prometheus.yml"

      - "--web.console.libraries=/etc/prometheus/console_libraries"

      - "--web.console.templates=/etc/prometheus/consoles"

      - "--web.external-url=http://mprometheus:9090"

    configs:

      - source: prometheus-3.0.3

        target: /etc/prometheus/prometheus.yml

      - source: alert-rules-3.1.4

        target: /etc/prometheus/alert.rules.yml

    labels:

      com.bnsf.mp.description: "Prometheus monitoring"

      com.bnsf.mp.department: "XF"

    networks:

      - monitoring

      - build

      - logging

    ports:

      - "9090:9090"

    volumes:

      - mprometheus:/mprometheus

    deploy:

      replicas: 1

      placement:

        constraints: [node.hostname == mwrk01]

      resources:

        limits:

          cpus: '4.0'

          memory: 4096M

      restart_policy:

        condition: on-failure

        delay: 5s

        max_attempts: 3

        window: 120s

      update_config:

        parallelism: 1

        delay: 10s

        failure_action: rollback

        monitor: 30s

        max_failure_ratio: 0.25

[rnm...@gmail.com]

Thanks for reply. I'm not using docker composer yet. :D (Still new to docker)

I guess if there isn't native method in prometheus, I'll try your configuration.

[Ben Kochie]

Prometheus 2.x has no memory limiting, because it doesn't have a separate cache memory like 1.x had.

It uses what it needs based on the metrics it is collecting, and the queries you are running.

Even in 1.x, the memory flag was not a limit, but a target. It would easily use more than the target limit.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4313fdef-68d6-4fb7-ac00-1a3801fdbdf7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rnm...@gmail.com]

I see. So what happens when I'm out of memory. Does Prometheus handle its memory usage or does it simply crash?

@ryans...@gmail.com How does your deployment works? I mean what happens when prometheus container hits this limit memory: 4096M?

[Ben Kochie]

It will crash.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/db5932b6-6a33-47e8-8add-de5aec7b880a%40googlegroups.com.