Pushgateway - any way to save/restore metrics

[Peter Toft]

Hi guys

I use a pushgateway -> prometheus -> grafana system to display simulation metrics, and when I reboot the server I loose all the metrics I have in the pushgateway. They appear when I have rerun my external test server pushing metrics.

If I know that there is a reboot of the pushgateway coming. Is there a simple'ish way to save and after a reboot restore these metrics?

Have a nice day

/Peter

[Stuart Clark]

The push gateway supports a persistence file via the --persistence.file parameter to allow cached metrics to survive restarts.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[Peter Toft]

Thanx Stuart

I use docker where I just now added       - '--persistence.file=/var/log/pushgateway.txt'
It does not seem to give persistence.

version: '3.3'

services:

  pushgateway:

    container_name: pushgateway

    image: prom/pushgateway

    ports:

      - 9091:9091

    volumes:

      - ./pushgateway.txt:/var/log/pushgateway.txt

    command:

      - '--web.enable-admin-api'

      - '--persistence.file=/var/log/pushgateway.txt'

  prometheus:

    container_name: prometheus

    image: prom/prometheus

    volumes:

      - ./prometheus.yml:/etc/prometheus/prometheus.yml

    ports:

      - 9090:9090

    command:

      - '--config.file=/etc/prometheus/prometheus.yml'

      - '--web.enable-admin-api'

      - '--storage.tsdb.retention.time=1y'

  grafana:

     container_name: grafana

     image: grafana/grafana

     volumes:

       - ./config/grafana.ini:/etc/grafana/grafana.ini

     ports:

       - 3000:3000

[Julien Pivotto]

You will need to use a docker volume to save the file across restarts.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/821d1de4-c59d-4cd9-938a-8e23dab2b2cbn%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Peter Toft]

Thanx for the comment Julien.

I try exactly this with 

 pushgateway:

    container_name: pushgateway

    image: prom/pushgateway

    ports:

      - 9091:9091

    volumes:

      - ./pushgateway.txt:/var/log/pushgateway.txt

    command:

      - '--web.enable-admin-api'

      - '--persistence.file=/var/log/pushgateway.txt'

but I tried to login to the docker instance - the /var/log/pushgateway.txt is empty :-(

[Julien Pivotto]

I think that this is flushed on disk at regular interval:
--persistence.interval

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a2359aa2-ba16-4862-9831-6aefb7f7d88dn%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Peter Toft]

The default is 5 minutes which is super for me. But I do not see anything happening

[Peter Toft]

Is anyone using this feature? Seems to be broken to mo and I cannot see any logs from this.

[Julius Volz]

Hi Peter, I just tried it locally with Pushgateway built from HEAD and the persistence across restarts works fine for me (pushed metrics are still present after a restart).

Maybe in your Docker setup the problem is that you are directly mounting in a single file, could you try passing the surrounding directory as the volume instead? See also https://forums.docker.com/t/sharing-a-single-file-via-a-volume-mount-does-not-work-as-expected/57088

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/dcecdd83-5fca-4fb5-a155-dbf477f1cce3n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Peter Toft]

Julius - THANX

I did exactly this, and it works!

I think I was tricked by docker and the access right to the directory. The volume I mount is now owned by the same user as the one running pushgateway. 

without chmod 777 or controlling the user access I could not write.

Super - I got it.

[Julius Volz]

Great!

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3f2cb90d-eca6-4eaa-9757-69b70c567f8fn%40googlegroups.com.